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Overview 


This report describes the implementation and early impacts of the Washington State Integrated 
Basic Education and Skills Training (I-BEST) program at three colleges: Bellingham Technical 
College, Everett Community College, and Whatcom Community College. I-BEST is a nationally 
known program that aims to increase access to and completion of college-level occupational 
training in a variety of in-demand occupational areas. Its signature feature is team teaching by a 
basic skills instructor and an occupational instructor during at least 50 percent of occupational 
training class time. Colleges operated I-BEST programs in one or more occupational areas 
including automotive, electrical, office skills, nursing, precision machining, and welding. I-BEST 
is one of nine career pathways programs being evaluated under the Pathways for Advancing 
Careers and Education (PACE) study sponsored by the Administration for Children and Families. 


The I-BEST program was launched in Washington in the 2006-07 academic year by the 
Washington State Board for Community and Technical Colleges. I-BEST aims to help students in 
basic skills programs (e.g., Adult Basic Education, English as a Second Language), who otherwise 
might have spent time in remediation, to enroll and succeed in college-level occupational 
training courses. Each I-BEST program is a course of study within a structured career pathway, 
and it offers students the opportunity to obtain credentials and college credits in in-demand 
occupations. Besides the team teaching, the I-BEST program evaluated in PACE also included 
two enhancements: financial support for tuition and associated materials; and additional 
advising services focused on supporting students’ academic needs, navigating college 
procedures, and career planning. These enhancements were funded by a grant to Abt 
Associates from the Open Society Foundations to support program modifications, and are not 
part of the standard I-BEST program at other colleges in Washington State. 


Using a rigorous research design, the study found that the I-BEST programs at the three colleges 
increased participation in college level courses, number of credits earned and credential 
attainment. Future reports will examine whether the I-BEST program resulted in gains in 
employment and earnings. 


Primary Research Questions 
e Was the intervention implemented as designed? 


e How did services received differ between study participants who could access I-BEST 
programs in the three colleges versus those who could not? 


e What were the effects of access to I-BEST on short-term education outcomes including 
earning credits and credentials? 
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Purpose 


Low-income workers with only a high school education or less face poor and declining 
employment prospects. Postsecondary training, often at community colleges, offers one 
strategy for improving this population’s education and employment opportunities, especially if 
it is targeted to occupations where there is high and growing demand for skilled workers. How 
to facilitate a better match between the nation’s need for a skilled workforce and the needs of 
low-skilled adults for employment is a topic of great interest to policymakers, workforce 
development organizations, educators, and other key stakeholders. 


Career pathways programs are designed to address these issues by providing well-articulated 
training and employment steps targeted to locally in-demand jobs, combined with a range of 
supports. Policymakers and practitioners have shown great interest in the career pathways 
approach. But, to date, limited rigorous research is available on its effects on participants’ 
educational and economic outcomes. To assess the effectiveness of a career pathways program 
such as I-BEST, the PACE evaluation uses an experimental design—that is, randomly assigning 


group 


|” 


study participants to a “treatment” group who can access the program and a “contro 
who cannot, then comparing their outcomes. 


Key Findings & Highlights 


e The three colleges implemented I-BEST as designed. The three colleges varied in how 
they delivered I-BEST across the different occupational programs, but implementation 
study findings suggest that the program was delivered largely as planned. The 
combination of the instructional approach (team teaching), advising, and financial 
supports resulted in a clear contrast between the services available to treatment group 
members and those available to control group members. 


e The program provided access to college-level courses for individuals with low basic 
skills and education levels. Almost one-third of treatment group members reported 
having less than a high school diploma or equivalent. Overall, 73 percent of treatment 
group members offered the I-BEST program participated in it. 


e The program had positive impacts on college course enrollment, driven primarily by 
enrollment in occupational training courses. I-BEST increased college enrollment by 22 
percentage points, and increased enrollment in occupational training courses by 41 
percentage points. 


e |-BEST increased credits earned and credential attainment. Treatment group members 
received an average of 13 more credits compared to control group members, and there 
was a 32-percentage point increase in credential receipt. 
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Methods 


The PACE evaluation’s implementation study examined the design and operation of I-BEST 
programs at three colleges and students’ participation patterns; its impact study used an 
experimental design to measure effects on educational and employment outcomes. The PACE 
evaluation pools results across the three colleges and all occupational training areas. 


From November 2011 to September 2014, the PACE evaluation randomly assigned 632 program 
applicants either to the treatment group (315 students), which could access the I-BEST 
program, or to the control group (317 students), which could not. Data were collected from 
college records for a 24-month follow-up period and a follow-up survey conducted 
approximately 18 months after random assignment. The evaluation also included site visits and 
monitoring calls to document program implementation and operations. Prior to estimating 
I-BEST impacts, the research team published an analysis plan specifying key hypotheses and 
outcome measures, and registered those outcomes.* 


1 Abt Associates. 2015. Pathways for Advancing Careers and Education (PACE). Technical Supplement to the 
Evaluation Design Report: Impact Analysis Plan. OPRE Report # 2015-100, Washington, DC: Office of Planning, 
Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human 
Services. https://www.acf.hhs.gov/opre/resource/pathways-for-advancing-careers-and-education- 


supplement-evaluation-design-impact-analysis-plan. 
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Executive Summary 


Low-skilled workers with only a high school education or less face poor and declining 
employment prospects. Postsecondary training, often at community colleges, offers one 
strategy for improving this population’s education and employment opportunities, especially if 
targeted to occupations where demand for skilled workers is high and growing. How to 
facilitate a match between the nation’s need for a skilled workforce and the needs of low- 
skilled adults for entry-level employment and advancement to higher-skilled jobs is a topic of 
great interest to policymakers, workforce development organizations, educators, and other key 
stakeholders. 


The Integrated Basic Education and Skills Training (I-BEST) Program 


This report describes the implementation and early impacts of the Washington State Integrated 
Basic Education and Skills Training (I-BEST) program at three colleges: Bellingham Technical 
College (BTC), Everett Community College (EvCC), and Whatcom Community College (WCC). 
Designed by the Washington State Board for Community and 


Technical Colleges (SBCTC), I-BEST operates at all 34 public Key Finding 

Within a 24-month follow-up period, 
treatment group members were 
provide occupational training and basic skills in a structured more likely than a randomly 
assigned control group who could 
not access the program to 


community and technical colleges in the state. It is designed to 


career pathway for students who have basic skills levels too 
low to enter college. e obtain workforce and academic 
: credits, and 
Without I-BEST, students whose college entrance test scores 

. . . i e receive a college credential. 
were too low to enroll directly in an occupational training 
program would first have to enroll in Adult Basic Education 
(ABE) or English as a Second Language (ESL) classes to raise their basic skills to the required 


levels. 
The key elements of the I-BEST program as defined by SBCTC are: 


e Focus on living wage jobs. |-BEST programs must incorporate coursework that will 
qualify workers for jobs with median wages of at least $13 per hour (or $15 per hour in 
King County, which includes Seattle). 


e Basic skills instruction through team teaching and support classes. I-BEST occupational 
training courses are required to have both an occupational instructor and a basic skills 
instructor present for at least 50 percent of class time. As a complement to team 
teaching, some I-BEST programs also include support classes, taught by basic skills 
instructors, that clarify concepts from occupational training classes, address basic skills 
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required to succeed in occupational classes, and help students prepare for assignments 
and tests. 


e Credits. Most I-BEST programs range in length from one to two quarters, and each 
quarter includes credit-bearing courses. Per SBCTC guidelines, workforce credits are 
earned in occupational training courses that focus on specific technical skills. For the 
most part, I-BEST generally did not result in academic credits, which are usually 
transferable to four-year colleges, including most general education requirements. 


e Credentials. Completion of I-BEST courses generally culminated in a workforce award 
from the college, although further state licensing may be required to practice in the field 
(e.g., for nursing occupations). 


e Subsequent training and credentials. As |-BEST programs are relatively short by 
definition, each program also defines a longer-term post-I-BEST educational pathway. 


Within this overall approach, individual colleges have flexibility in how they design and operate 
their I-BEST program, including the occupations targeted. 


In addition to these standard I-BEST elements, the three colleges in the PACE evaluation 
received additional resources for two program enhancements funded by the Open Society 
Foundations: 


e A dedicated advisor available to I-BEST students to assist with guidance on academic 
issues, navigating the college’s procedures, and career planning. 


e “Fill-the-gap” financial support for tuition and course-associated materials that could 
not be covered by other sources. 


Pathways for Advancing Careers and Education (PACE) Evaluation 


Abt Associates and its partners are evaluating I-BEST as part of the Pathways for Advancing 
Careers and Education (PACE) evaluation. Funded by the Administration for Children and 
Families (ACF) within the U.S. Department of Health and Human Services, PACE is an evaluation 
of nine programs that include key features of a career pathways framework. The framework 
guides the development and operation of programs aiming to improve the occupational skills of 
low-income, low-skilled adults, primarily older nontraditional students, by increasing their entry 
into, persistence in, and completion of postsecondary training. Central to accomplishing these 
outcomes, the framework describes strategies for overcoming barriers to education and 


2 All community and technical colleges governed by SBCTC provide workforce awards (a certificate or an 
Associate in Applied Sciences degree requiring more 20 or more credits that are not transferable to a four- 
year college); workforce completion (a certificate requiring less than 20 credits); and associate’s degrees that 
require two years of coursework and are transferrable to a four-year college. 
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training that these students can face. Key features of programs within this framework include: a 


series of well-defined training steps, promising instructional approaches targeted to adult 


learners, services to address academic and non-academic barriers to program enrollment and 


completion, and connections to employment. 


The I-BEST evaluation includes an implementation study that examines the design and 


operation of the program and enrolled students’ participation patterns. It also includes an 


impact study that measures differences in education and employment outcomes using an 


experimental design; that is, by comparing individuals randomly assigned to a treatment group 


that could enroll in I-BEST at the three colleges versus a control group that could not.? 


The three community and technical colleges in the 
study offer I-BEST programs in a different set of 
occupational fields (see box). The colleges were 
selected due to the size of their I-BEST program and 
willingness to participate in a random assignment 
study. They were not selected to be representative of 
the I-BEST program across all community and technical 
colleges in the State of Washington. The PACE 
evaluation of I-BEST pools results across the programs 
in the three colleges. 


Analyses used data from two baseline surveys 
completed at the time of study entry, an 18-month 
follow-up survey, 24 months of college records, and site 
visits. This report provides the results from the 


Colleges and I-BEST Programs 
Included in PACE Evaluation 


Bellingham Technical College 

e Automotive 

e = Electrical 

e Nursing Assistant 

e Precision Machining 

e §©Welding 

Everett Community College 

e Nursing Assistant 

e Sustainable Office Skills 

e §©Welding 

Whatcom Community College 
Clerical Administration 


implementation study and describes the early impacts of the I-BEST program (18 to 24 months 


after random assignment) on education, training receipt, credits earned (the confirmatory 


outcome selected to assess the early effects of I-BEST), and credential receipt, as well as early 


employment outcomes. Though there are previous evaluations of I-BEST, PACE is the first to use 


a rigorous random assignment research design to determine a causal link between access to I- 


BEST and education outcomes. 


3 Random assignment ensures that the treatment and control groups will be alike in their observed and 
unobserved characteristics, and that any systematic differences in their outcomes can be attributed to the 


treatment group having access to program services. 
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Findings Summary 


From the Implementation Study 


e I-BEST served individuals with low education and basic skills levels, including 31 percent who had 
less than a high school diploma or equivalent. 


As discussed, I-BEST seeks to increase access to college-level classes for students with low basic 
skills. Reflecting this approach, the high proportion of study participants who lacked a high 
school diploma or equivalent reflects the low basic skills level of the population served. More 
than two-thirds of study participants reported never having attended college, and less than 10 
percent reported having an associate’s degree or higher. I-BEST allowed students with lower 
skills levels to enroll in the college-level occupational training programs that included basic skills 
instruction. 


e Though the details varied across the colleges and I-BEST programs, the three colleges 
implemented I-BEST team teaching by combining basic skills and occupational technical content. 


A hallmark of the I-BEST model is team teaching. As noted above, basic skills instructors must 
be present for at least 50 percent of occupational training class time. The colleges and 
instructors had flexibility in how they approached team teaching, including how occupational 
and basic skills instructors defined their roles within the classroom and planned their time 
together. The implementation study documented three general approaches: 


1. Basic skills instructor as an active student, sitting in class with the students and 
stopping the occupational instructor to ask clarifying questions or for a concept 
to be further explained if students were having difficulty with the material. This 
approach was used often in occupational courses that were primarily lecture- 
based and highly technical, such as Welding and Machining. 


2. Basic skills instructor delivers part of the content during a designated part of the 
occupational classes. For example, the basic skills instructor might start each 
class with a discussion of study skills before turning it over to the occupational 
instructor to present technical information. 


3. Basic skills and occupational instructors jointly deliver class content. From the 
students’ perspective, delivery is seamless; it may not be apparent which 
instructor is responsible for which type of content. Staff reported this approach 
was more suitable to I-BEST programs where the occupational focus was not 
highly technical or specialized, such as Office Skills. 
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e The I-BEST teaching teams reported benefiting from experience in working together, but some 
faced challenges in finding adequate instructional planning time. 


Two colleges had teaching teams that worked together for multiple quarters. These teams 
tended to have clearly defined roles and often co-delivered content. I-BEST staff reported that 
new teams had a learning curve. For many occupational instructors, I-BEST courses were their 
first experience teaching low-skilled students, whereas basic skills instructors needed to learn 
occupational content quickly. Regardless of team tenure, college staff reported that some 
teaching teams had limited preparation time due in part to the adjunct status of instructors. 


e Basic skills instructors provided advising and support. 


Basic skills instructors at each of the three colleges provided tutoring and one-on-one academic 
support, especially early in the academic term when students struggled with course material. 
Though each of the three colleges offered tutoring services to all students, including those in I- 
BEST, staff reported that students were often hesitant to engage with tutors and more inclined 
to seek the assistance of the basic skills instructors whom they already knew. 


e The I-BEST programs provided “fill-the-gap” financial assistance and dedicated student advisors 
(“navigators”) with funding from the Open Society Foundations. * 


Many I-BEST students were new to college and had never completed the Free Application for 
Federal Student Aid (FAFSA) or applied for other forms of financial aid. Navigators helped new 
students apply for financial aid for which they were eligible. The program then provided “fill- 
the-gap” funding if the financial aid was not sufficient. Colleges also used the funding to cover 
training-related expenses such as books. 


Navigators also provided proactive advising to I-BEST students regarding class progress, 
potential barriers to participation, and career and academic planning. Instructors and 
administrators at one college characterized the navigator as a “one-stop shop” who served as 
both a coordinator and guide for I-BEST participants. Another administrator described the 
navigator role as an “early alert system” to proactively identify potential issues that could 
inhibit student success.° 


4 Programs in PACE could receive funding from OSF to enhance program services. OSF made a grant to Abt 
Associates for this purpose, and funds were allocated by Abt to programs that proposed modifications. 

5 Each of the colleges had advising services available to I-BEST students prior to the start of PACE. However, 
these services were less comprehensive than what was available to treatment group members during the 
study. Feedback from staff from Washington’s State Board for Community and Technical Colleges indicates 
that more comprehensive navigation services are now a core component of I-BEST. 
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e The I-BEST program did not offer structured employment services, but some instructors provided 
informal employment and job search assistance. 


SBCTC designed the I-BEST program with an explicit focus on occupational training that could 
increase the earnings potential of participants. However, none of the three colleges had 
employment supports specific to |-BEST. Administrators at one college noted that the lack of 
employment services was a weakness of the I-BEST program, whereas another administrator 
indicated that the lack of such specific employment services reflected the focus on continued 
training and credentials. 


Although none of the three colleges had dedicated I-BEST employment services, staff reported 
that instructors often provided informal, individualized assistance to students as requested. 
Some I-BEST instructors came from the relevant industry and knew employers in the local labor 
market. One Welding instructor, for example, stayed informed of employers who were likely to 
hire entry-level welders and provided applications to students and coached them on the 
application process. 


e Recruiting students for I-BEST was challenging. 


For the PACE study, the three colleges collectively planned to enroll 1,000 study participants to 
be equally divided between the treatment and control groups. Based on past experience, 
college administrators expected a high demand for occupational training among their currently 
enrolled basic skills students (ABE and ESL), and several I-BEST programs included in the study 
traditionally had waiting lists. In addition, colleges expanded their outreach to community- 
based organizations, employers, the state’s Temporary Assistance for Needy Families (TANF) 
program, and the local media. They also reached out to students whose college entrance scores 
were high enough to enroll in the standard college occupational classes but whose basic skills 
levels were in the I-BEST range. Despite efforts to increase recruitment, the three colleges 
struggled to recruit participants for the study and ultimately recruited two-thirds of the sample 
goal (632 study participants in total). 
e Across the three colleges, almost three-quarters of treatment group members participated in an I- 
BEST program. 
Seventy-three (73) percent of treatment group members participated in at least one I-BEST 
course. Exhibit ES-1 shows the proportion of the treatment group that participated in an I-BEST 
course and, for those who participated, the percentages who obtained a credential and 
enrolled in additional courses after I-BEST within a 24-month follow-up period. 


Abt Associates Executive Summary | pg. ES-vi 


Washington State’s Integrated Basic Education and Skills Training (I-BEST) Program in Three Colleges 
Implementation and Early Impact Report 


PACE 
Exhibit ES-1: Participation in I-BEST, Credential Receipt, and Participation in Post-I-BEST 
Courses, by College and Program, within 24 Months after Random Assignment 
Participated in an Obtained Workforce _ Enrolled in Additional 
I-BEST Program Credential Courses after I-BEST 
Among Those Among Those Who Participated ina 
College/Program Randomly Assigned I-BEST Program 
Bellingham Technical College 
Automotive (N=3) 100% 33% 67% 
Electrical (N=13) 87% 31% 77% 
Nursing Assistant (N=57) 71% 70% 53% 
Precision Machining (N=6) 60% 83% 50% 
Welding (N=39) 81% 38% 72% 
Subtotal (N=118) 76% 55% 62% 
Nursing Assistant (N=20) 61% 80% 55% 
Sustainable Office Skills (N=36) 86% 61% 67% 
Welding (N=22) 48% 95% 55% 
Subtotal (N=78) 65% 76% 60% 
Clerical Assistant (N=38) 89% 29% 76% 
Total (all colleges and programs) 73% 58% 63% 


SOURCE: SBCTC college records. 
NOTES: Sample size for column one is 315; sample size for columns two and three is 230 and includes all individuals who 
attended at least one team-taught I-BEST program. 


As shown in the first column, almost 90 percent of WCC treatment group members enrolled in 
an I-BEST course, whereas 76 percent of BTC students and 65 percent of EvCC students did so. 
Participation rates were above 80 percent for the Automotive, Electrical, and Welding programs 
at BTC; the Sustainable Office Skills program at EvCC; and the Clerical Assistant program at 
WCC. The lowest rate (48 percent) was in the I-BEST Welding program at EvCC. 


e Fifty-eight (58) percent of treatment group members who participated in I-BEST obtained a 
workforce credential, and almost two-thirds enrolled in subsequent education and training. 


Each college awarded a college-issued certificate for the completion of its |-BEST programs. As 
shown in the second column on Exhibit ES-1, 58 percent of treatment group members who 
participated in I-BEST received a credential within the 24-month follow-up period, with the 
majority being workforce awards (20 credits or more; not shown). 


Credential receipt varied by college and occupational area. More than three-quarters of EvCC 
students (76 percent) obtained a credential, compared with 55 percent of BTC students and 29 
percent of WCC students. Credential receipt rates were above 80 percent for treatment group 
members who enrolled in the BTC’s Precision Machining and EvCC’s Nursing Assistant and 
Welding programs. EvCC had notably higher credential receipt rates than comparable programs 
at the other colleges for students enrolling in its Welding program (95 percent for EvCC versus 
38 percent for BTC) and in its clerical program (61 percent for EvCC versus 29 percent for WCC). 
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The third column of Exhibit ES-1 shows that a high proportion of treatment group members 
who participated in an I-BEST program continued their education beyond I-BEST, with 63 
percent enrolling in additional college courses after I-BEST. This rate was similar at BTC and 
EvCC (about 60 percent) but higher at WCC (76 percent). On average, those who attended an I- 
BEST program were enrolled in college courses for four quarters—longer than the I-BEST 
portion of the occupational training lasted. Approximately one-quarter were still enrolled in 
college at the end of the 24-month follow-up period (not shown). 


The variation in credential receipt, as well as the progression to additional education and 
training (in some cases, a greater proportion than those who received a credential through 
I-BEST), is likely due to I-BEST students not completing all of the I-BEST program course 
requirements but continuing to higher or perhaps different courses of study. 


From the Impact Study 


e  |-BEST had a positive impact on the number of academic and workforce credits earned at colleges 
(confirmatory outcome). 


As Exhibit ES-2 shows, I-BEST had a 13-credit impact on academic and workforce credit 
completion, which was the confirmatory outcome for the impact analysis. Both types of credits 
are college-level credits (i.e., non-remedial and applicable towards credentials), and the 
accumulation of these credits is a positive indicator of academic progress, either towards a 
credential or, for academic credits, towards transferring to four-year programs in the future. 
However, the majority of the credits (84 percent) earned were workforce credits (not shown). 


The impact on credits earned was driven primarily by more treatment group members enrolling 
in college than control group members, and to a lesser extent by treatment group members 
who attended college earning more credits than control group attendees. These results indicate 
engagement in college-level courses for many of these students would not have otherwise 

been possible, because they did not meet the college entry basic skill requirements. 
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Exhibit ES-2: Early Impacts on Credits Earned and Credential Receipt, within 24 Months after 
Random Assignment (Confirmatory and Secondary Outcomes) 


Total number of academic \ 
1S .1*** 


and workforce credits earned 
(confirmatory outcome) 


Received a certificate or degree Ne \* 


4.2 
from a college (secondary outcome) 
Credential Receipt by Type 


8.5 
Workforce completion [2 3.7** 


B 47 
or 333 


Workforce award 79) 


Associate’s degree or higher ie 0.0 


@ Treatment Group (§ Control Group 
*** statistically significant at the one percent level; ** at the five percent level. 


SOURCE: SBCTC records. 
e |-BEST produced large positive impacts on credential completion, particularly workforce awards. 


As Exhibit ES-2 shows, I-BEST increased the completion of any credential from a SBCTC college 
within 24 months after random assignment by more than 32 percentage points, with more than 
44 percent of the treatment group receiving some type of credential within the 24-month 
follow-up period compared with only 12 percent of the control group (based on SBCTC college 
records). The majority of the students completed a workforce award, a categorization that 
includes either a certificate that required more than 20 total workforce credits to complete or 
an Associate’s in Applied Science (AAS) degree. I-BEST also increased by four percentage points 
completion of short-term workforce completion certificates, which includes any certificate that 
requires less than 20 total workforce credits to complete. 


The impact on credentials completed was driven primarily by a higher proportion of treatment 
group members who enrolled in college receiving a credential than did the control group 
members who enrolled in college. Among those who enrolled in college, 54 percent of 
treatment group members received a credential from a college, whereas 28 percent of control 
group members did so (not shown). 
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e  |-BEST had large positive impacts on college course enrollment, driven primarily by enrollment in 
occupational training courses within the first six months after random assignment. 


As Exhibit ES-3 shows, I-BEST increased college enrollment by 22 percentage points, with close 
to 90 percent of the treatment group enrolling in college compared with 68 percent of the 
control group. As shown in the exhibit, I-BEST increased enrollment in occupational training 
courses by 41 percentage points (81 percent for treatment group members compared with 40 
percent for control group members over the 24 months—a 100 percent increase). The large 
impact on occupational training course enrollment is driven by treatment group members 
enrolling in |-BEST courses, whereas the control group members would likely take remediation 
classes first. The treatment group members could earn workforce credits for completing I-BEST 
courses. I-BEST also increased academic course enrollment by six percentage points. 


Exhibit ES-3: Early Impacts on Enrollment in Occupational Training and Academic Courses, 
within 24 Months after Random Assignment 


89.6 


% Enrolled in any courses 


cany month) 677 a 
% Enrolled in academic 31.8 61** 
courses (any month) 25.6 : 
6 Enrolled moccupations — 
training courses (any month) 


@ Treatment Group e Control Group 


*** statistically significant at the one percent level; ** at the five percent level. 
SOURCE: SBCTC records. 


The impact in occupational training class enrollment is at its largest in the first six months after 
random assignment, with 73 percent of the treatment group enrolling in an occupational 
training course compared with only 22 percent of the control group (not shown). The impacts 
on cumulative occupational training course enrollment decrease over time, where eventually 
the treatment and control groups are roughly equal in months 19-24 after random assignment 
(not shown). 


e There was a positive impact on the enrollment and completion of college-level algebra and English 
courses. 


As Exhibit ES-4 shows, I-BEST increased enrollment in and completion of college-level algebra 
and English, both of which are credit-bearing academic courses and are often considered to be 
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“gateway” courses for advancement into higher-level college courses. For algebra, I-BEST 
increased enrollment by 10 percentage points and completion by nine percentage points; for 
English, I-BEST increased enrollment by nine percentage points and completion by six 


percentage points. These findings are encouraging in terms of advancement in college 


coursework, since these algebra and English courses serve as prerequisites for many other 
courses required as part of two-year associate degrees and are often transferrable to four-year 


colleges. 


Exhibit ES-4: Early Impacts on the Enrollment and Completion of College-Level Algebra and 


English 


Enrollment and completion of 
college-level academic courses 


% Enrolled in introductory 
college-level math 


% Enrolled in introductory 
college-level English 


% Completed introductory 
college-level math with 
a grade of C or better 


% Completed introductory 
college-level English with 
a grade of C or better 


10.3$*** 


9.3 


| 


@ Treatment Group @§ Control! Group 


*** statistically significant at the one percent level; ** at the five percent level. 


SOURCE: SBCTC records. 


e  |-BEST had statistically significant impacts on receipt of advising and employment services. 


Reflecting the dedicated advisor that was part of I-BEST, the program had a 14 percentage point 
impact on receipt of career counseling (36 percent of the treatment group members versus 22 
percent of control group members, statistically significant at the one percent level). There were 
similar impacts on receipt of help arranging supports for school, work, or family (29 percent of 
the treatment group versus 16 percent of the control group, statistically significant at the one 
percent level). Though there were no official job search assistance services, |-BEST produced an 
impact on receipt of job search or placement services (29 percent of the treatment group 
versus 19 percent of the control group, statistically significant at the one percent level), 


potentially reflecting the assistance received from instructors or the advisor. 
(See full results in Chapter 4, Exhibit 4-5). 
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Implications 


Because of the interest at the federal, state, and local levels in the I-BEST as a strategy to 


improve education and employment outcomes for low-skilled adults, replications of the model 


are already underway in several states and localities. These results have a number of 


implications for further development of related initiatives. 


Initial findings from the PACE evaluation provide strong causal evidence of the effect 
the I-BEST program can have on education outcomes. The PACE evaluation provides 
the first rigorous evidence on I-BEST from a range of occupational programs at three of 
the state’s 34 community and technical colleges. Though not representative of all I-BEST 
programs statewide, these results greatly strengthen the evidence that I-BEST can 
produce increases in receipt of credits and credentials. Future reports will provide 
information on employment effects at later follow-up points. 


Though the I-BEST programs themselves last one to two quarters, many students used 
them as a launching pad for additional college-level education and training. A high 
proportion of treatment group members who attended I-BEST programs continued their 
education beyond I-BEST, with 63 percent enrolling in additional college courses after 
I-BEST. On average, those who attended an I-BEST program were enrolled in college 
courses for four quarters—longer than the I-BEST portion of the occupational training 
lasted. Approximately one-quarter were still enrolled in college at the end of the 
24-month follow-up period. 


Maintaining a connection between occupational and academic programs may be 
important for students interested in pursuing higher levels of education that may 
result in higher-paying jobs. Most of the credits earned and credentials received by 
I-BEST participants were for workforce rather than academic courses, meaning they are 
not transferable to four-year colleges. But there is some evidence of students moving to 
academic courses of study. Primarily through enrollment in courses after I-BEST, the 
program increased enrollment in and completion of college-level algebra and English, 
both of which are credit-bearing academic courses and are often required as part of 
two-year associate degrees and transferrable to four-year colleges. 


The team teaching approach benefited from planning and durability in teaching 
teams. Staff reported variations in how team teaching was operationalized and 
challenges the teams faced. To effectively implement I-BEST the instructors reported 
that they needed to adapt to the learning styles of students with lower basic skills as 
well as develop instructional approaches that effectively integrate a second instructor. 
Overall, the study indicates the value of planning activities to help instructors define and 
integrate their instruction and importance of the tenure of the instructors in working 
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together as team. In addition, several of the programs experienced challenges using 
staff who were part-time and adjunct, which is common in community college 
programs, because it limited the planning time available to the teaching teams. 


e Although the I-BEST programs operated largely as designed, the implementation 
experience showed areas for further attention and development. First, although the 
program sought to target low-skilled individuals, colleges had some difficulty recruiting 
students. Second, connections to employment and jobs were not a formal component 
of the program in most of the career areas. Finally, study findings on the impacts of 
student advising and supports indicate that the dedicated student advisors and “fill-the- 
gap” financial assistance—resources not typically available to I-BEST students but 
available during the study through a PACE-related foundation grant—may be a 
contributing factor to the program effects observed. 


Next Steps 


The next I-BEST report will cover a 36-month follow-up period. It will focus on the effects of the 
program on students’ economic outcomes for the period when these are expected to occur. 
The report will examine employment outcomes, such as average employment and earnings 
over successive follow-up quarters, and job characteristics. Thus, it will begin to answer 
whether the services provided by I-BEST translate into economic gains in the workplace in the 
longer term. An analysis at 72 months after random assignment will estimate long-term effects 
on earnings of the program. 


The many important questions that remain to be addressed in subsequent reports include: 


e Will |I-BEST’s impact on educational attainment remain stable, increase, or decrease? 


e Will |-BEST’s impact on educational attainment translate into impacts on employment 
and earnings? 


e Does I-BEST have other impacts on participants and their families? 


e Is |-BEST cost beneficial? 
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Workers with only a high school education or less face poor and declining employment 
prospects (Georgetown University Center on Education and the Workforce 2016; Pew Research 
Center 2014). Postsecondary training, often at community colleges, offers one strategy for 
improving this population’s employment opportunities, especially if it is targeted to 
occupations where demand for skilled workers is high and growing (Capelli 2014; Conway and 
Giloth 2014; Holzer 2015). How to meet the nation’s need for a skilled workforce and the needs 
of low-income adults for entry-level employment and advancement to higher-skilled jobs is a 
topic of great interest to policymakers, workforce development organizations, educators, and 
other key stakeholders. 


Research indicates that meeting both needs is not easy. Many low-income, low-skilled adults 
face considerable barriers to completion of postsecondary education. Many are 
“nontraditional” students—that is, often they are older, are parents, lack adequate basic 
academic skills, and have few economic resources (National Center for Education Statistics 
1996). Further, on average, nontraditional students fare worse in postsecondary settings than 
do traditional students (Visher et al. 2008; Cooper 2010; Goldrick-Rab and Sorensen 2010). 
Institutions often assign students who need to improve their basic academic skills to 
developmental (remedial) education, courses that do not result in college credits. Many of the 
students requiring remediation never progress beyond it as the amount of remedial coursework 
makes the transition to college-level coursework difficult (Bailey, Jeong, and Cho 2010; 
Rutschow and Schneider 2011). Others drop out due to financial setbacks or difficulties juggling 
school, work, and family responsibilities. Some have difficulties navigating the college 
environment, including course sequences and financial aid (Karp 2011). Many have difficulty 
meeting academic standards (Bridges to Opportunity Initiative 2008). 


This report describes the implementation and early impacts of the Washington State 
Integrated Basic Education and Skills Training (I-BEST) program at three colleges: Bellingham 
Technical College, Everett Community College, and Whatcom Community College. I-BEST is a 
nationally known program that aims to increase students’ access to and completion of college- 
level occupational training in a variety of in-demand occupational areas. 


I-BEST aims to quickly teach students basic, occupational, and college-readiness skills so they 
can move through college and into living wage jobs faster. The program uses a team teaching 
approach. Students work with two instructors in the classroom: one provides job-training and 
the other teaches basic skills in reading, math, or English (SBTC 2017). 
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Abt Associates and its partners are evaluating the I-BEST program as part of the 10-year 


Pathways for Advancing Careers and Education (PACE) evaluation.® The evaluation includes 


both an implementation study to examine I-BEST’s design and operation and an impact study to 


estimate the impacts of access to I-BEST on its students’ education and training, employment, 


and other outcomes within an 18- to 24-month follow-up period.’ Although PACE is not the first 


study of I-BEST’s impacts (Zeidenberg, Cho and Jenkins 2010; Jenkins, Zeidenberg, and Kienzl 


2009), it is the first to use a rigorous random assignment research design. 


This chapter describes the PACE evaluation, provides an overview of the I-BEST program, 


summarizes findings from previous I-BEST evaluations, and provides a roadmap to the rest of 


the report. 


1.1. 


PACE is funded by the by the Administration for 
Children and Families (ACF) within the U.S. 
Department of Health and Human Services. It is the 
first large-scale, multi-site experimental evaluation 
of programs that include key features of a “career 
pathways framework.” This framework articulates 
signature strategies for overcoming the barriers that 
nontraditional, occupational students often face. For 
example, strategies typical of programs within this 
framework include having a series of well-defined 
training steps, promising instructional approaches, 
supportive services, and connections to employment 
(Fein 2012). The career pathways framework is 
flexible, however, and not a specific program model. 
Thus, which components a local program adopts and 
how it implements them can vary greatly. 


Reflecting this diversity, each of the nine programs 
in the PACE evaluation represents a different 
program model (see box). All share some program 


Pathways for Advancing Careers and Education (PACE) Evaluation 


Programs in PACE 


Bridge to Employment in the Health Care Industry 
at San Diego Workforce Partnership, San Diego, 
CA 


Carreras en Salud at Instituto del Progreso Latino, 
Chicago, IL 

Health Careers for All at Workforce Development 
Council of Seattle-King County, Seattle, WA 


Integrated Basic Education and Skills Training (I- 
BEST) program at three colleges (Bellingham 
Technical College, Everett Community College, 
and Whatcom Community College), Washington 
State 

Pathways to Healthcare at Pima Community 
College, Tucson, AZ 

Patient Care Pathways Program at Madison 
College, Madison, WI 

Valley Initiative for Development and Advancement 
(VIDA), Lower Rio Grande Valley, TX 

Workforce Training Academy Connect at Des 
Moines Area Community College, Des Moines, IA 
Year Up (Atlanta, Bay Area, Boston, Chicago, 
National Capital Region, New York City, 
Providence, Seattle) 


§ For more information on the PACE study, go to www.acf.hhs.gov/opre/research/project/pathways-for- 


advancing-careers-and-education. 


7 The time frame was selected because completion of the 15-month follow-up survey occurred on average 18 
months after random assignment and college records were available for the full sample for 24 months post 


random assignment. 
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components that are part of the career pathways framework; but each also has distinct and 
unique elements, reflecting the target populations, occupational trainings offered, and 
industries of focus. Because of this variation, PACE evaluates and reports findings for each 
program individually.® 


The central goal of the PACE evaluation is to determine the effectiveness of each of the nine 
programs using a common evaluation design and conceptual framework (impact study). The 
most critical element of the evaluation design is random assignment of eligible applicants 
either to a treatment group that can access the career pathways program or to a control group 
that cannot. A randomized control design ensures that the study’s treatment and control 
groups will be equivalent in their observed and unobserved characteristics—and thus, that any 
systematic differences in their subsequent outcomes can be attributed to the treatment group 
having access to program services (i.e., the program’s impacts). Systematic differences in 
outcomes due to the characteristics of individual members in each group can be ruled out. 


Consistent with this career pathways framework and the career pathways theory of change 
(described in Chapter 2) guiding the PACE evaluation, the key outcomes for which the PACE 
study estimates effects are in two areas—education and training and employment—although 
the study also estimates effects in other areas, such as individual and family well-being. 


The PACE evaluation implementation and early impact program reports analyze outcomes over 
a follow-up period of about 18 to 24 months after random assignment. The impact analyses rely 
primarily on surveys and college records for study participants in the treatment group and 
control group. Future reports will analyze outcomes three years and six years after random 
assignment.? These future reports will also include benefit-cost studies for some of the nine 
PACE programs. 


1.2. Overview of the I-BEST Model 


Developed by the Washington State Board for Community and Technical Colleges (SBCTC), the 
I-BEST program has been operating in all 34 public community and technical colleges in the 
state since the 2006-07 academic year. The I-BEST model grew out of a concern that adult basic 
skills students—adults who lack high school-level skills—were not advancing beyond basic skills 
classes to college-level occupational programs, and therefore were not earning credentials. In 
Washington State, adult basic skills programs that are offered at two-year colleges, including 


8 All PACE-related reports can be found on www.career-pathways.org as well as 
www.acf.hhs.gov/opre/research/project/pathways-for-advancing-careers-and-education. 


9 These future reports will be developed as part of the Career Pathways Intermediate Outcomes and the Career 
Pathways Long-term Outcomes studies, respectively. 
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English as a Second Language (ESL) and Adult Basic Education (ABE), serve approximately 
60,000 students per year (Wachen, Jenkins, and Van Noy 2010). 


One influential study (Prince and Jenkins 2005) found that only 13 percent of students who 
enrolled in ESL programs and 30 percent of students who entered ABE programs continued on 
to earn any college credits. Very few (four to six percent of each group) earned the credit 
equivalent of two full-time semesters or a certificate within five years. However, ESL and ABE 
students who did complete a year of credits and a credential earned on average $7,000 and 
$8,500 per year more, respectively, than those who did not. This one-year threshold of credits 
and credential completion is known as the “tipping point,” a marker by which students are 
more likely to have successful longer-term outcomes such as employment. However, there is no 
rigorous evidence that indicates that the tipping point causes these outcomes to occur. Perhaps 
it is the more determined and academically ready students who were already likely to have 
successful outcomes who reach this tipping point. 


SBCTC funded a pilot at 10 of Washington’s community and technical colleges in the 2004-05 
academic year to increase the rate of transition of ABE and ESL students to college-level 
occupational training programs. It rolled out the program to all 34 colleges over the next two 
academic years. 


Core Components of the I-BEST Standard Model 
SBCTC created the I-BEST model with the following core components: 


e Courses are part of a structured career pathway, defined by SBCTC as “a sequence of 
courses that leads directly to a postsecondary credential and to jobs that are in demand 
in the local labor market.” Pathways ensure students do not have to “find their way on 
their own” (SBCTC 2005). 


e Ateam teaching instructional model pairs basic skills instructors and occupational 
instructors for at least 50 percent of occupational training class time. 


e Enhanced funding from SBCTC reimburses colleges 1.75 times the regular rate for a full- 
time equivalent (FTE) student to help cover the costs associated with implementing I- 
BEST, including development of a curriculum, instructor preparation, and supportive 
services such as a dedicated program coordinator. 


Within this |-BEST model, the colleges have flexibility in creating a new program. A college must 
complete a comprehensive application that details the local demand for that specific 
occupational field, as well as a roadmap to the full educational pathway for that program. 
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PACE Enhancements to the I-BEST Model 

In addition to these core components, the three colleges in the PACE study received additional 
funding for program enhancements from the Open Society Foundations. The enhancements 
were available only to I-BEST students while they were members of the PACE study’s treatment 
group: 


e Dedicated advising. Each college had a dedicated advisor (“navigator”) to provide 
students with guidance on academic issues, navigating the college’s procedures, and 
career planning. 


e “Fill-the-gap” financial support for training and associated materials beyond typical 
sources available to any student. For students who were not able to secure funding 
through Pell Grants, Washington State Opportunity Grants, or other sources, the three 
colleges covered the tuition costs. They also provided funds for books, tools, other 
course materials, or transportation. 


Replication of I-BEST 

There have been several efforts in other states and localities to replicate the I|-BEST program or 
some of its key components such as team teaching. For example, Maryland has implemented 
the I-BEST model in some of its community colleges, and Mississippi has implemented it 
statewide.!°11 Other efforts include the Breaking Through program, which works with 41 
community colleges in 22 states.’ Like I-BEST, Breaking Through provides low-skilled students 
with accelerated, contextualized training in in-demand fields, plus comprehensive supports. 


Yet another effort to replicate the I-BEST model, Accelerating Opportunity, operates at the 
state-system level rather than the college level. It aims to transform basic skills education in five 
states./3 The program emphasizes support through a “navigator” or “coach” more than I-BEST 
itself does. A recent study in four of the five states used non-experimental design to determine 
impacts of Accelerating Opportunity on academic and labor market outcomes (Anderson et al. 
2017). It found mixed evidence on the effectiveness of this program, with only one of the four 
study states found to have mostly positive impacts for both types of outcomes. Rigorous 
research documenting replications of I-BEST has not yet been released. 


Prior Research on I-BEST 


Previous study of I-BEST suggests that the program can improve outcomes such as credential 
and credit completion. Two often cited studies from the Community College Research Center 


10 See http://insidemc.montgomerycollege.edu/details.php?id=25781. 


1 See http://news.hindscc.edu/index.php/hinds-cc-mi-best-program-focus-of-hearing-before-lawmakers/. 


2 See http://www. jff.org/initiatives/breaking-through 


3 See http://www. jff.org/initiatives/accelerating-opportunity. 
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(CCRC) used system-wide administrative data from SBCTC and non-experimental quantitative 
methods to determine impacts, but they are not as rigorous as a well-implemented randomized 
study. 


Zeidenberg, Cho, and Jenkins (2010) estimated the impact of I-BEST among basic skills students 
who had enrolled in at least one career or technical education course on their own. Using the 
same administrative data collected from SBCTC on individual students, their transcripts, and 
their credentials, the authors used two different methods—a difference-in-differences approach 
in addition to a separate propensity score matching approach—to determine the impact of I- 
BEST on outcomes. This research found that I-BEST had a positive impact on college credit 
accumulation and on basic skills, and mixed findings on credential completion. These results 
replicated largely similar findings from an earlier study (Jenkins, Zeidenberg, and Kienzl 2009). 
Neither study found a positive impact on wages nor hours worked, which may have been due to 
students entering the job market during a major recession. 


Though these two studies used rigorous, non-experimental quantitative methods, the resulting 
impact estimates might suffer from selection bias — the more motivated students may have 
elected to enroll in I-BEST. This possibility highlights the potential contribution of new evidence 
based on random assignment, which protects against selection bias. 


1.3. Research Context for Key Features of I-BEST 


Though previous evidence on the impact of I-BEST on outcomes is limited, numerous studies 
exist on its major components. 


Improving Basic Skills 

There is considerable evidence that the usual approach of requiring remediation before 
allowing students to enroll in college-level coursework does not help them complete college 
coursework. Most students referred to developmental education to raise their skills to college 
standards never enroll in college-level courses (Bailey, Jeong, and Cho 2010). Research using 
the National Education Longitudinal Survey found that fewer than half of students referred to 
developmental coursework even complete the required sequence of developmental courses 
(Bailey et al. 2010). However, there is little evidence about whether the alternative that I-BEST 
offers is more effective. I-BEST is an acceleration strategy that compresses the time required to 
succeed in an occupational program by simultaneously providing basic skills training and 
occupational training. There is some evidence that compressing developmental education into 
shorter periods can improve outcomes for low-skilled students, but there is no rigorous 
evidence of its effectiveness (Rutschow and Schneider 2011). 
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Contextualization 

I-BEST delivers basic skills content in a contextualized manner, using materials and examples 
from the student’s occupational field of interest. Advocates for contextualization argue that it 
can improve student motivation and success, because students may be more likely to engage 
with basic skills materials that use occupational subject matter that interests them. The 
evidence on the effectiveness of contextualization among adult learners is mixed, with very few 
randomized studies at the college level and with some studies finding limited effects (Perin 
2013; 2011). However, non-experimental evidence does suggest the possibility of impacts on 
college readiness and progress toward vocational certificates (Bragg and Barnett 2009). 


Enhanced Advising 

The I-BEST model at the three colleges included more individualized advising and navigational 
support than is typically available at two-year colleges in Washington State. This advising was 
designed to ease students’ transition to college-level classes and to proactively address any 
academic or non-academic barriers that emerged while students were enrolled. 


Evidence from random assignment evaluations has demonstrated that enhanced counseling 
and more personalized advising can increase the number of courses passed and credits 
completed (Bettinger and Baker 2011; Scrivener and Weiss 2009.) The City University of New 
York’s Accelerated Study in Associate Programs (ASAP) has had a strong and convincing impact 
on associate’s degree completion, and the program strongly emphasizes advising. However, it is 
not certain whether the advising is responsible for these results, as it is only one component of 
the program (Scrivener et al. 2015). Some other programs in PACE, notably Valley Initiative for 
Development and Advancement (VIDA), also strongly emphasize advising among other 
components; early results from VIDA are promising (Rolston, Copson, and Gardiner, 2017). 
Project QUEST, a program similar to VIDA that provides comprehensive support through 
community colleges to help individuals complete occupational training programs, has also seen 
promising results through a randomized research design (Elliott and Roder 2017). 


Added Financial Support for Training and Related Costs 

In their CCRC study, Jenkins, Zeidenberg, and Kienzl (2009) found that I-BEST students were 
more likely to receive financial aid than were students enrolled in career and technical 
education classes but not in I-BEST. During the PACE study, the program at the participating 
colleges provided additional financial supports over and above those provided by Washington 
State that were funded by the Open Society Foundations (OSF), to ensure that lack of financial 
aid would not be a barrier to enrollment or program completion.** A large body of evidence 
indicates that insufficient resources are a barrier to entry and completion of education and 


4 Programs in PACE could receive funding from OSF to enhance program services. OSF made a grant to Abt 


Associates for this purpose, and funds were allocated by Abt to programs that proposed modifications. 
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training for low-income students, and that financial assistance can increase postsecondary 
attendance and persistence (Deming and Dynarski 2010; Dynarski and Scott-Clayton 2013). 


Similarly, OSF funds covered costs associated with transportation, books, and other school 
supplies, as well as testing fees and certification exams. The Opening Doors Demonstration, 
which had positive effects on student outcomes, included financial supports as part of the 
program model (Scrivener and Weiss 2009), though the contribution of those financial supports 
to the program’s positive effects is uncertain. The Performance-Based Scholarship 
Demonstration (Mayer et al. 2015) included a range of academically incentivized financial aid 
supports and had modestly positive effects on students’ academic progress and degree 
completion outcomes. 


1.4. Structure of This Report 


The organization of the remainder of this report follows: 


e Chapter 2 presents the PACE evaluation as applied to I-BEST, beginning with its 
conceptual framework and research questions; details the evaluation design; describes 
the study sample; and summarizes the evaluation’s data sources. 


e Chapter 3 describes the context and design of I-BEST in the three colleges. 


e Chapter 4 describes the implementation study findings, including training programs and 
instructional approaches, participation in training and comparisons of participation in 
education and training across the treatment and control groups, academic and 
nonacademic advising, employment supports, and financial assistance provided by the 
program. 


e Chapter 5 presents the impact study’s early findings, focusing on two main impacts— 
hours of training, and credentials received over a 24-month follow-up period—but also 
looking at other career and life outcomes. 


e Chapter 6 summarizes the implementation and impact findings and discusses their 
implications for the PACE study in the longer term. 


The appendices provide additional details about baseline data (Appendix A); college records 
data, a key source for the impact study (Appendix B); survey-based outcomes (Appendix C); and 
the approach to outliers (Appendix D). 


15 See https://www.mdrc.org/sites/default/files/designing scholarships FR.pdf. 
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2. PACE Evaluation Design and Data Sources 


This chapter describes the larger PACE evaluation design and its application to I-BEST. It begins 
with a discussion of the PACE career pathways theory of change and the research questions 
that the theory of change implies. It then briefly describes the evaluation design and analysis 
procedures for the impact study, including the random assignment process and the outcome of 
that process. A brief description of the implementation study analysis follows.*¢ Finally, the 
chapter summarizes the main data sources for the implementation and impact studies. 


2.1. Career Pathways Theory of Change 


The career pathways theory of change guides both the implementation study (that is, it 
identifies which aspects of program services are expected to affect outcomes) and the impact 
study (that is, it identifies which outcomes the program is expected to affect). The theory of 
change also generates key hypotheses about the direction of expected effects that the impact 
evaluation will test for statistically significant change.?’ In addition, the theory of change 
implicitly assumes time horizons by which the program is expected to have effects, and thus the 
theory determines the key outcomes at any particular time of follow-up. 


Exhibit 2-1 depicts the PACE career pathways theory of change, as applied to I-BEST in the three 
participating colleges. It shows how a program (inputs) is hypothesized to produce effects on 
intermediate outcomes, which in turn will lead to effects on main outcomes. Effects on 
intermediate outcomes are expected to occur earlier than effects on main outcomes, but the 
exact timing depends on particular features of the program, such as the length of occupational 
training and what, if any, steps precede it. In addition, because effects on intermediate 
outcomes may persist over time, the study will also measure them at later points in time. 


16 The research team developed a detailed evaluation design report and analysis plan for the PACE evaluation 


(Abt Associates 2014, 2015). 


The implementation study describes the set of services that students in the treatment group experienced. In 
addition to descriptive statistics, it includes a small number of impact estimates that show the difference in 
services received between treatment and control group members. The impact study focuses solely on 
estimates of the effects of the program on intermediate and main outcomes. 


17 
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Exhibit 2-1: Career Pathways Theory of Change for I-BEST 


PROGRAM INPUTS 


fo} -te7 Nira Nate), | 


* Bellingham Technical 
College 

* Everett Community 
College 

* Whatcom Community 
College 

*I-BEST program 
standards and funding 
set by SBCTC 

*I-BEST plan for each 
college 

* Open Society 
Foundations funding 


PARTICIPANTS 


+ Students scoring in 
each college's specified 
range on the CASAS 
test 

+ Students’ interest in 
occupational areas 
offered by colleges’ 
|-BEST programs 


PROGRAM COMPONENTS 


ASSESSMENT 
* CASAS 


INSTRUCTION 


* Basic skills and training 
in occupational skills 
programs 

* Team teaching 

« Instructor-led study 
group, open lab, 
tutoring 


Abt Associates 


SUPPORTS 


* Career and academic 
advising and counseling 
by dedicated program 
advisors 

+ Fill-the gap financial 
support: 

- Tuition for training 
- Tools, books, and 
transportation 


EMPLOYMENT 


* College career services 
* Clinical placements 
(for Nursing Assistant) 
+ Internships 
(for Sustainable Office 
Skills) 


INTERMEDIATE OUTCOMES 


GENERAL (21ST CENTURY) COMPETENCIES 


+ Improved basic academic skills 


+ Improved psycho-social skills (persistence, 
academic self-confidence, self-evaluation, 
sense of belonging) 


SPECIFIC COMPETENCIES 


«Improved skills in desired occupational area 


CAREER KNOWLEDGE 
+lncreased awareness of steps needed to reach 
career goals 


+ Understanding of required steps needed to 
reach career goals 


+ Increased knowledge of labor market 


RESOURCES 


* Barriers to entry and completion of 
occupational training addressed through 
advising, financial assistance, and concurrent 
support for basic skills and English language 
skills 


LIFE CHALLENGES 


« Reduced financial hardship 
* Reduced stressors 


MAIN OUTCOMES 


POSTSECONDARY ATTAINMENT 


* Hours of training received 
* Credits 
* Credentials 


SUCCESSFUL IN CAREER-TRACK 
EMPLOYMENT 


* Obtain employment in occupations aligned 
with I-BEST training programs 

+ Increased earnings and job benefits 

+ Perceived career progress 


OTHER LIFE OUTCOMES 


«Improved individual well-being 
+ Improved family economic status 


CONTEXTUAL FACTORS 


LOCAL POSTSECONDARY 
TRAINING SYSTEMS 


* Job openings in occupations 
aligned with I-BEST training population 
programs 


* Competing training programs 


LOCAL ECONOMY 


OTHER COMMUNITY FACTORS 


* Size and characteristics of target 


* Supportive service providers 
¢ Referral partners 
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As shown in Exhibit 2-1, starting in the box at the left, the career pathways theory of change 
begins with two types of program inputs: ?® 


Organization. Organizational inputs include the three colleges, program standards and 
funding from SBCTC, and supplemental funding from the Open Society Foundations. 


Participants. This individual input includes the characteristics of the target population, 
such as ABE and ESL students scoring in the specified range on the Comprehensive Adult 
Student Assessment System (CASAS) assessment and their interest in an occupational 
area offered by a college’s I-BEST program. 


This same box includes four types of program components that are expected to improve 


participant outcomes by addressing barriers that are hypothesized to impede successful entry 


into and completion of occupational training: 


Assessment. |-BEST programs administer the CASAS to assess English and math skill 
levels and to determine whether the applicant is eligible for |-BEST. +9 


Supports. |-BEST supports include career and academic advising by dedicated program 
advisors, and “fill-the-gap” financial support for tuition (not covered by financial aid) 
and for training-related tools, books, and transportation. 


Instruction. |-BEST integrates basic skills and occupational training. Courses are team 
taught—basic skills instructors and occupational training instructors must be in the 
training classroom together at least 50 percent of the time. Some programs also offer 
instructor-led study groups, open labs, or tutoring. 


Employment. |-BEST students could access their college’s general career services and 
career counselors. Two of the I-BEST programs included in the study, Nursing Assistant 
and Sustainable Office Skills, made placements in clinical or internship settings. 


The middle box shows intermediate outcomes—targeted improvements expected to lead to 


better main outcomes. These include improved basic academic skills; improved psycho-social 


skills such as grit and academic self-confidence; improved skills in occupational areas; and 


increased career knowledge. 


18 


19 


Program inputs can include components available only to treatment group members as well as those available 
to both treatment group and control group members, because the interaction of the former components with 
the latter can lead to impacts. 


Entities such as colleges, employment programs, and employers use CASAS to assess basic skills, literacy, and 
English language skills need for both academic and workplace success. 
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In the far right box, the main outcomes are the primary targets that I-BEST seeks to change: 


e Increased postsecondary attainment, namely accumulated hours of training and credits 
(as measures of progress toward a credential) and occupational training credentials. 


e Successful employment, including obtaining employment in occupations that align with 
I-BEST training programs and pay at least $13 per hour ($15 per hour in King County), 
increased earnings and job benefits, and career advancement. 


e Improvements in individual and family finances and well-being. 


Influencing expected effects are a number of contextual factors, including the types and 
number of postsecondary training systems in the local area and strength of the local economy. 
Other community factors are the size and characteristics of the target population, and the 
number and nature of other service providers. 


2.2. Research Questions for the I-BEST Evaluation 


The implementation study documented I-BEST in the three colleges as implemented and 
captured participation patterns of treatment group members in training (see Chapter 4 for 
implementation findings). The impact study (see Chapter 5) aimed to measure the effectiveness 
of I-BEST in improving students’ short-term (within 18-24 months) intermediate and main 
outcomes. 


Implementation study research questions: 


e What is the intended program model? What is its institutional and community context? 


e What intervention was actually implemented? Did it deviate from plans or 
expectations? 


e What were the treatment group’s participation patterns and experiences with program 


services? 


e What are the differences in services, including training, received by treatment and 
control group members? 


Impact evaluation research questions: 


e What were the main effects of I-BEST on: 


— Educational attainment, including hours of occupational training received, credits 
received, credentials received, and other education outcomes? 


— Entry into career-track employment, higher-wage jobs, earnings, and perceptions of 
career progress? 
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— Participant and family well-being, including income and material hardship? 


e To what degree did the program affect intermediate outcomes in the theory of change, 
such as: 


— Confidence in career knowledge and access to career supports? 


—  Psycho-social skills such as grit, academic self-confidence, core self-evaluation, and 
social belonging at school? 


— Life stressors, such as financial hardship, life challenges, and perceived stress? 


As discussed, the program’s theory of change not only describes hypothesized causal 
connections, it also identifies time horizons over which they are expected to occur. For 
example, with respect to I-BEST, the students’ first step along their chosen career pathway is an 
initial occupational certification that typically takes one to two academic quarters at the college 
to complete. Program completers can look for employment commensurate with the 
certification or continue on the next step of the pathway. Those who continue their education 
after I-BEST enroll in coursework for more advanced certifications (e.g., one lasting three 
quarters) or for an associate’s degree. Given that some advanced courses of study are longer 
than others, it is possible that some students will still be enrolled in training at the end of the 
24-month study period. Thus, this early impact report focuses primarily on training outcomes 
that do not require a study participant to complete a program and earn a credential. 


For this early impact report, the primary data sources for addressing the impact research 
questions are SBCTC records of course enrollment, completion, and credentials; two surveys 
administered at “baseline” (when students joined the study and were randomly assigned to the 
treatment or control group); and a follow-up survey of all study participants initiated at 
approximately 15 months after random assignment. The implementation study questions are 
addressed with information gathered during site visits and monitoring calls. A more complete 
description of data sources is in the concluding section of this chapter. 


Later PACE reports will focus more on employment outcomes and on outcomes resulting from 
education and training activities that require longer to complete (e.g., associate’s degree 
programs). The continued measurement of such outcomes will be important, given that the 
career pathways framework implies that workers may alternate between education and 
employment as they move along a pathway. 


2.3. PACE Evaluation Design and Analysis 


As mentioned in Chapter 1, the PACE evaluation uses a random assignment research design to 
estimate the impact of access to a program on students’ outcomes. The great benefit of such a 
design is that when properly implemented, it ensures that estimated effects reliably can be 
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attributed to access to the program—and not to unmeasured differences in characteristics or 
external circumstances between the treatment group (who can access the program) and the 
control group (who cannot). 


However, maintaining the comparability of the treatment and control groups requires 
comparing all of those in the treatment group with all of those in the control group, regardless 
of whether or not treatment or control group members actually enrolled in the program (what 
researchers refer to as an “intent to treat” analysis). A critical implication of this is that the 
evaluation estimates the impact of access to the entire program—to the entire I-BEST program, 
in this case—as opposed to the impact of the program’s specific components. The evaluation 
does so by comparing the entire control group with the entire treatment group, regardless of 
the treatment group’s actual take-up of one or many program components. 


A second feature of the PACE impact study design is that both treatment and control group 
members can access education, training, and support services available in the community that 
are not exclusive to the program PACE is evaluating. In the case of I-BEST, the evaluation 
estimates the effect of the program’s components above and beyond what was otherwise 
available at the three colleges and elsewhere in their communities during the study period. For 
example, both treatment and control group members could potentially access their college’s 
occupational training courses that were not part of I-BEST, subject to availability and meeting 
entry requirements. Thus, the control group’s experiences represent what would have 
happened absent I-BEST. 


In summary, the impact study assessed whether the existence of this multi-component career 
pathways program led to better outcomes for students who were offered the chance to 
participate, given what these students could have obtained without the program. ”° 


2.3.1. College Selection, Study Intake, and Random Assignment 


The PACE evaluation of I-BEST is designed to pool results across the three colleges in the study 
and their various I-BEST occupational programs. The PACE research team began recruiting 
colleges for the I-BEST study in 2010. The team sought to recruit large I-BEST programs at 
multiple colleges, with an ultimate goal of enrolling at least 2,000 study participants across the 
colleges. Over the course of a few years, the team contacted colleges across the state, had 


20 Four technical appendices provide additional details about analysis methods. Appendix A describes data 


collected at baseline, gives further detail on baseline characteristics of treatment and control group members, 
and explains procedures for using these data to adjust for imbalances arising by chance during random 
assignment. Appendix B describes college records data serving as the main source for measuring program 
impacts on educational progress. Appendix C provides detail on survey-based outcome measures, adjustments 
for item nonresponse, and analyses of survey nonresponse. Finally, Appendix D documents the research 
team’s approach to outliers. 
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discussions with program administrators to learn more about the I-BEST programs (e.g., 
historical and projected enrollment, presence of waitlists, interest in participating in an 
experimental evaluation) and conducted site visits to the most promising eight community or 
technical colleges. Five of these colleges ultimately declined to participate in PACE. Reasons for 
not moving forward included concern about their ability to over recruit for a control group and 
resistance to changes in their enrollment processes to accommodate the evaluation. 
Bellingham Technical College (BTC), Everett Community College (EvCC), and Whatcom 
Community College (WCC) agreed to participate. However, because two of these colleges had 
small I-BEST programs, the evaluation team reduced the target sample size to 1,000 study 
participants. 


The research team worked with each college to design and implement study procedures and 
enhancements to the standard I-BEST model using OSF funds (see Chapter 1). The research 
team and program staff developed research procedures that allowed for a rigorous test of I- 
BEST while minimizing the effect of the study on normal college operations. Additionally, the 
team provided ongoing technical assistance to the colleges to support recruitment and 
enrollment of participants into the study. Each college designated key staff, typically the I-BEST 
navigator, to serve as the primary liaison throughout implementation of the study. 


Interested program applicants took the CASAS math and reading assessments to determine 
whether they were eligible for the I-BEST program of interest.?7 Eligible applicants then met 
with I-BEST staff to confirm their interest in the program and to address any program-specific 
eligibility requirements (e.g., Nursing Assistant applicants were screened for tuberculosis). 


Program staff then enrolled eligible applicants into the study (intake): 


1. Informed Consent. Program staff discussed the PACE evaluation and offered its 
informed consent form. Applicants who declined to sign the informed consent form 
were excluded from the study and could not enroll in the I-BEST program(s) at the 
colleges that were part of the PACE study. Those who consented became study 
participants and proceeded to the next step. 


21 ‘The colleges varied slightly in required CASAS reading and math scores. At BTC and WCC, students were 
required to score between 221 (4th to 5th grade equivalent) and 256 (12th grade equivalent) on both tests to 
be eligible for I-BEST. The exception was the Nursing Assistant program at BTC, which required only 211 on 
math. EvCC required students to score at least 201 on the math and reading CASAS tests, but typically 
recommended that students should score above 211 before enrolling in I-BEST. 
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2. Baseline Data. Staff asked study participants to complete the study’s Basic Information 
Form (BIF) and Self-Administered Questionnaire (SAQ). The BIF collected demographic 
and economic information. The SAQ measured a variety of attitudes, beliefs, and 
psycho-social dispositions, as well as more sensitive personal characteristics such as 
perceived ability to stay calm in stressful situations. Again, applicants who declined were 
excluded from the study and the I-BEST program(s). 


3. Random Assignment. Program staff used an online system to randomly assign study 
participants to either the treatment group or the control group. The random assignment 
ratio was 1:1, so that the treatment and control groups would each include 
approximately half of the study sample. 


4. Services According to Random Assignment Status. Study participants assigned to the 
study’s treatment group could access the I-BEST program(s) at their college. Under the 
evaluation’s “intent-to-treat” design (discussed earlier), they were included in the 
analysis whether or not they actually enrolled in I-BEST training. Participants assigned to 
the control group could not access the I-BEST program(s) at their college. Both 
treatment and control group members could access comparable other services available 
in the community, including those offered by the same college in which I-BEST was 
operating. 


Between November 2011 and September 2014, program staff randomly assigned 632 study 
participants: 315 to the treatment group and 317 to the control group. Exhibit 2-2 shows the 
distribution by college.*? Random assignment occurred separately at each college, but all three 
colleges followed the same general procedures. 


Exhibit 2-2: Number of Study Participants Randomly Assigned, by College 


College Treatment Control Total 
Bellingham Technical College (BTC) 156 159 315 
Everett Community College (EvCC) 121 120 241 
Whatcom Community College (WCC) 38 38 76 
Total 315 317 632 


22 Random assignment at BTC occurred between November 2011 and September 2014. Random assignment at 
EvCC occurred between February 2013 and April 2014. Random assignment at WCC occurred between 
October 2012 and April 2014. 
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2.3.2. Characteristics of the Study Sample 


Exhibit 2-3 shows the percentage distributions of the treatment and control group members 
across a Set of characteristics. The p-values in the last column test the hypotheses that there 
are no systematic differences between the groups for these characteristics.7° 


As shown, random assignment produced treatment and control groups without significant 
differences in observed baseline characteristics with three exceptions: age, food assistance 
receipt, and public assistance receipt. These differences are most likely due to chance,’ given 
the number of characteristics tested (see Hypothesis Testing section below). In conducting 
impact analyses, the research team controls for any bias resulting from these and other 
differences by using baseline values as covariates to adjust for chance differences (described in 
the Impact Estimation section below). 


Exhibit 2-3 also shows the composition of the study sample. Participants have low levels of 
education, with 31 percent reporting less than a high school diploma or equivalent. Less than 
10 percent reported having attended one or more years of college. 


Sample members are low income, with a mean annual income of $22,110 and 47 percent 
reporting income of less than $15,000. Consistent with these low levels of income, 59 percent 
of study participants received benefits from the Supplemental Nutrition Assistance Program 
(SNAP, formerly known as Food Stamps) or Special Supplemental Nutrition Program for 
Women, Infants, and Children (WIC) in the 12 months prior to study intake. About 21 percent of 
study participants received public assistance or welfare (Temporary Assistance for Needy 
Families) in the prior year. Receipt of food assistance and public assistance in the previous 12 
months was somewhat higher in the treatment group than in the control group. 


Many study participants were older than traditional college students. More than 60 percent 
were older than age 25, although 22 percent of the sample was age 20 or younger. There was a 
statistically significant difference in ages between the treatment and control groups at the 10 
percent level. The majority of study participants (58 percent) were female. Slightly more than 
half (55 percent) were non-Hispanic white, and about one quarter (26 percent) identified as 
Latino or Hispanic. Two-thirds (67 percent) were not working at the time of random 
assignment, with only 13 percent working 35 hours or more. 


23 The p-value summarizes results from chi-squared tests of the likelihood that the difference in the observed 
value or larger would occur if there were no differences between the two samples. For example, a p-value of 
.32 means that even if the characteristics of the members in the treatment and control groups were identical, 
the observed difference or larger would occur 32 percent of the time. 


4 See Appendix A.2. On average, one would expect that three out of 28 variables would have statistically 
significant mean differences. Regression adjustments also help to control for any effects that chance 
differences might have on the impact estimates. 
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Exhibit 2-3: Selected Characteristics of the I-BEST Study Sample 
All Study Treatment Control 
Characteristic Participants Group Group p-Value 
Age (%) .067* 
20 or younger 22.2 23.2 21.1 
21 to 24 14.9 1441, 18.6 
25 to 34 29.8 31.4 28.1 
35 or older 33.2 34.3 32.2 
Gender (%) 231 
Female 57.5 55:1 59.9 
Male 42.5 44.9 40.1 
Race/Ethnicity (%) 346 
Hispanic 26.0 28.9 23.1 
Black, Non-Hispanic 7.6 6.2 9.1 
White, Non-Hispanic 54.9 53.1 56.7 
Other, Non-Hispanic 14.1 13.4 148 
Family Structure (%) 591 
Not Living with Spouse/Partner and Not Living with Children 47.2 48.7 45.8 
Not Living with Spouse/Partner But Living with Children 16.6 14.6 18.6 
Living with Spouse/Partner and Not Living with Children 17.3 18.2 16.3 
Living with Spouse/Partner and Children 18.9 18.5 19.3 
Living with Parents 28.6 27.2 30.1 412 
Current Education (%) 497 
Less Than a High School Diploma or Equivalent 30.7 28.2 33.1 
High School Diploma or Equivalent 40.0 42.0 38.0 
Less Than 1 Year of College 11.1 12.1 10.2 
1 or More Years of College 9.5 10.1 8.9 
Associate’s Degree or Higher 8.8 tal 9.8 
Income (%) 051 
Less than $15,000 47.3 46.5 48.1 
$15,000-$29,999 23.9 26.0 21.9 
$30,000 or More 28.8 27.6 30.0 
Mean $22,110 $23,002 $21,240 378 
Public Assistance/Hardship in Past 12 Months (%) 
Received WIC or SNAP 58.6 55.0 62.1 .092* 
Received Public Assistance or Welfare 21.3 18.1 24.3 .094* 
Reported Financial Hardship 48.5 49.8 47.1 499 
Current Work Hours (%) 993 
0 66.6 66.9 66.3 
1to 19 8.5 8.5 8.5 
20 to 34 11.7 11.7 11.6 
35 or more 13.2 12.8 13.6 
Sample Size 631 315 316 


SOURCE: PACE Basic Information Form. 

SNAP is Supplemental Nutrition Assistance Program. WIC is Special Supplemental Nutrition Program for Women, Infants, and Children. NOTE: * 
denotes statistically significant differences at the p = .10 level. Appendix A provides a fuller set of baseline characteristics random assignment 
generated well-balanced treatment and control groups. Some percentages do not add up to 100.0% due to rounding. Public Assistance/Hardship 
in Past 12 Months does not add to 100% because the categories are not mutually exclusive and exhaustive. 
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2.3.3. Analysis Plan for the Impact Study 


The PACE evaluation of I-BEST is designed to pool results across the three colleges in the study 
and their various I-BEST occupational programs. Because only three of Washington State’s 34 
community and technical colleges are included in the study, the results should not be 
considered representative of all community and technical colleges in the state (see box). 


Three of 34 Community and Technical Colleges 


The three colleges included in the PACE study were not randomly selected and are not 
representative of community and technical colleges in Washington. Nor are the college I- 
BEST programs representative of I-BEST programs implemented across the state. However, 
the three colleges do reflect the diversity of student populations and approach to program 
delivery. As shown below, there is variation across the colleges in terms of overall student 
population size, size of |-BEST programs, and the percentage of students who receive need- 
based financial aid. 


The three colleges fall across the distribution of SBCTC colleges in terms of overall enrollment 
and receipt of need-based financial aid. However, none of the three colleges had an 
especially large I-BEST program: BTC’s was roughly average and EvCC and WCC had smaller 
than average I-BEST programs. Chapter 3, Section 3.1 provides more details on the local 
context of the three colleges. 


Student Headcount 
in Need-based Financial 
School Overall Enrollment I-BEST (#) Aid Receipt (%) 
Bellingham Technical College 5,926 125 54 
Everett Community College 19,388 73 21 
Whatcom Community College 11,292 22 46 
SBCTC Average 10,581 126 38 


https://www.sbctc.edu/about/facts-publications/field-quide-2017.aspx 
https://www.sbctc.edu/resources/documents/colleges-staff/research/academic-year-research/2015-16/ayr-1516-3-selected-programs.pdf 
Data from 2015-2016 Academic Year 


Prior to estimating I-BEST impacts, the research team published an analysis plan (Abt Associates 
2015) specifying key hypotheses and outcome measures and registered the plan on the What 
Works Clearinghouse and Open Science Framework websites.?? The team subsequently 


25 The analysis plan was posted to the What Works Clearinghouse (WWC) online registry of randomized control 


trials (RCT) on May 20, 2016. In September 2016, under the terms of a grant from the Institute of Education 
Sciences, the RCT registry information was removed from the WWC website and transferred to the Society for 
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assessed data quality, refined the plan for I-BEST, and publicly registered it on the Open Science 
Framework website.*° The purpose of the analysis plan and registration was to guide the work 
of the research team and publicly commit to particular hypotheses and an estimation approach, 
in alignment with ACF’s commitment to promote rigor, relevance, transparency, independence, 
and ethics in the conduct of evaluations.’’ Pre-specification and registration help to establish 
the scientific rigor of research by documenting that inspection of early results did not influence 
the selection of findings in PACE reports. 


Hypothesis Testing 

An essential principle in the PACE analysis plan is to organize and discipline the number of 
statistical tests conducted. Like most social policy evaluations, the nine PACE studies target an 
array of different outcomes. If the evaluation did not adjust in some way for multiple 
hypothesis tests, a potentially large number of the tests would reach conventional levels of 
Statistical significance by chance, even if there were no effect on any outcome. This is known as 
the problem of “multiple comparisons.” To address this issue, the team established three 
categories of hypotheses: confirmatory, secondary, and exploratory: 


e Confirmatory hypotheses involve outcomes most critical to judging whether the 
program seems to be on track—that is, producing the results expected at a given 
follow-up duration. Given the relatively small sample sizes in the PACE studies, the 
research team limited such tests to one per program in the early impact report (at 18-24 
months after randomization for I-BEST) and two tests in each subsequent report (at 
three and six years after random assignment)—selecting outcome(s) under the “main” 
category in the program’s theory of change (see Exhibit 2-1). Each confirmatory 
hypothesis has an expected direction of change, an increase or decrease in the 
outcome. Therefore, the research team tests each confirmatory hypothesis for 
significance only in the specified direction, ignoring possible effects in the other, by 
applying a one-tailed test of statistical significance. 


e Secondary hypotheses involve a set of additional indicators consistent with expected 
effects within the period covered by the study report. As with the confirmatory 
hypothesis, each secondary hypothesis has an expected direction of change, and 
therefore, the research team tests each secondary hypothesis for significance only in 
the specified direction by applying a one-tailed test of statistical significance. Secondary 


Research on Educational Effectiveness (SREE). SREE plans to re-launch the registry in late 2018, at which time 
the analysis plan will be available in a searchable online database. 


26 See https://osf.io/6n5ua/. 


27 See https://www.acf.hhs.gov/opre/resource/acf-evaluation-policy. 
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analyses for I-BEST included tests of hypotheses for additional education outcomes, as 
well as a number of indicators of early career progress. 


e Exploratory hypotheses cover an additional set of possible effects that still fit within the 
framework of the theory of change, but whose direction and timing are less certain. 
Accordingly, the team applies two-tailed tests to these hypotheses to account for the 
uncertainty in the direction of impact. For this report, the research team selected 
several indicators of psycho-social outcomes (e.g., grit) and education success (e.g., 
enrolled in developmental math courses). 


Chapter 5 identifies the specific hypotheses in each category tested for I-BEST. 


Impact Estimation 

Random assignment ensures that, on average, samples of treatment and control group 
members will have similar characteristics at the outset. As a result, measured differences in 
outcomes provide unbiased estimates of program impacts. Following common conventions, the 
team compensated for chance differences in measured baseline characteristics when 
estimating program outcomes. Doing so also helps to increase the precision of estimates. 


For PACE, the research team estimated a statistical model relating each outcome to baseline 
variables for the control group sample, then calculated average differences between actual and 
predicted values in both groups and subtract the control group average from the treatment 
group average to generate the impact estimate. Appendix A provides a detailed description of 
this method. 78 


The research team estimated this approach both for continuous outcomes (e.g., total college 
credits earned) and for binary outcomes (e.g., yes/no questions). For survey-reported 
outcomes, weights were used to average outcomes. Additional details can be found in the 
technical appendices. 


Formally, estimation uses the following equation: 


where 6 is the estimated impact of being in the treatment group (whether or not the person 
attended the program or used any of the offered services); Y is the observed outcome of 


interest (e.g., credits); Y isa prediction of Y based on baseline variables measured at random 


28 As explained in section A.3 of Appendix A, the approach is a variant on the traditional approach to regression- 
adjustment methods used in impact analyses. The latter typically involves linear regression of each outcome 
on an indicator of treatment status and a series of baseline variables. In this approach, the coefficient on the 
treatment indicator provides the regression-adjusted impact estimate. 
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assignment; T is an indicator of treatment status (which is set equal to 1 if the individual is 


assigned to the treatment group and 0 if the individual is assigned to the control group); n, 


and n,. are the respective sample sizes in the treatment and control groups; and the subscript / 


indexes individuals. 


2.3.4. Analysis Plan for the Implementation Study 


The PACE evaluation’s implementation study relies on both qualitative and quantitative 


analyses, as well as a broad variety of data sources. Key analyses include the following: 


2.4. 


Descriptive. Describing each program’s design and context and developing its theory of 
change relied primarily on review of program materials (e.g., prior evaluations of I-BEST, 
college-specific documentation on the components of the I-BEST program); in-person 
discussions with program staff and leadership during two rounds of site visits; and 
biweekly or monthly calls between study and program leadership during the study 
period when random assignment was ongoing. 


Quantitative. A quantitative analysis of the proportion of program participants who 
reached major program milestones served to systematically document their experience 
in the program. This relied on college records and follow-up surveys of treatment and 
control group members. 


Fidelity. The study relies primarily on field observations and interviews with program 
staff and participants to assess the delivery of the I-BEST program. To address the 
question of how program delivery changed over time, the research team asked program 
staff about internal or external obstacles and how staff altered the program in an 
attempt to overcome them. Quantitative analysis of how and the extent to which 
students moved through the program also enables the comparison of the actual delivery 
of the program versus its design. 


Service Differences. Because the random assignment design of the impact study 
implicitly ensures that any effects of the program result from the different experiences 
of treatment and control group members, a key task of the implementation study is to 
describe the difference in services the two groups received. This is particularly 
important for the PACE evaluation, as the control group is not barred from receiving 
services similar to those the study’s treatment group received. 


Data Sources 


The PACE evaluation’s implementation and impact studies use a variety of data sources. 


Baseline Surveys. Prior to random assignment into the evaluation, program applicants 
completed two baseline surveys: The Basic Information Form (BIF) collected 
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demographic and economic information. The Self-Administered Questionnaire (SAQ) 
measured a variety of attitudes, beliefs, and psycho-social dispositions, as well as more- 
sensitive personal characteristics. 


e Follow-Up Survey. The research team sought to survey all study sample members 
starting at 15 months after random assignment. On average, the survey occurred 18 
months after random assignment. The survey asked questions on participants’ training 
and service receipt, postsecondary educational attainment, employment, income, debt, 
and participation in income support programs. It used a mixed-mode approach, 
conducted initially by telephone and then in person for those participants not reached 
by telephone. The I-BEST study collected surveys from 238 treatment group members 
and 225 control group members, yielding response rates of 76 percent and 71 percent, 
respectively.? 


e Administrative Records. The research team relied on the administrative records of each 
program evaluated, both to describe the experience of treatment group members in 
their program and to estimate program effects. The I-BEST study used records from 
SBCTC to measure treatment and control group members’ participation in education 
and training, as well as their credential receipt. 


e National Student Clearinghouse (NSC). The NSC is a national repository of information 
that captures the vast majority of college enrollments at public and private nonprofit 
institutions. The I-BEST study used data from the NSC to impute outcomes for 
participants who attended colleges not part of the SBCTC system. 


e Site Visits and Monitoring Calls. For the implementation study, the research team made 
two site visits to each of the three colleges. The goal of the first round of visits was to 
document the program’s theory of change and key components and to assess 
implementation of evaluation procedures. The goal of the second round of visits was to 
document any modifications to operations or the provision of services, as well as 
implementation challenges and plans for sustaining the program beyond the study 
period.°° During both sets of visits, the research team interviewed program managers; 
staff involved in evaluation activities (e.g., recruitment, intake, random assignment); 
instructors; advisors; case managers; and staff at any partner agencies with an 
important role in service delivery. In addition to these visits, the research team had 
regular conference calls with program staff during the random assignment period to 
discuss program updates, recruitment activities, intake and random assignment 


29 See Appendix C for response bias analyses. 


30 Visits to BTC occurred in March 2012 and November 2014. Visits to EvCC occurred in August 2013 and July 
2014. Visits to WCC occurred in May 2013 and April 2014. 
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processes and any challenges, engagement in the program by treatment group 
members, and staffing changes. 


e Program Documents. The research team obtained and reviewed program documents 
including funding applications; course catalogs; program materials such as applications, 
assessment tools, syllabi, and pre-training checklists; annual reports; and reports to 
funders. 
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The context in which I-BEST operates and its design at the three colleges participating in PACE 
are important to understanding its implementation and potential impacts. This chapter begins 
with a description of the local context for each of three colleges during the PACE evaluation 
enrollment period (November 2011 to September 2014). Next, it describes the comparable 
services available at the colleges and in the communities and expected treatment differential. 
Finally, it describes the I-BEST program design, course offerings, and instructional and support 
services.?? 


3.1. Local Context 


Three aspects of the local environment are important to understanding I-BEST’s 
implementation and impacts—the characteristics of the population in the area served by each 
college, the local labor market, and the services and opportunities available to control group 
members. 


3.1.1. Population and Labor Market 


As discussed in Chapter 2, of Washington State’s 34 community and technical colleges, three 
took part in the PACE evaluation of |-BEST—Bellingham Technical College (BTC), Everett 
Community College (EvCC), and Whatcom Community College (WCC). Exhibit 3-1 below shows 
the location of each, as well as the occupational courses of study for I-BEST programs included 
in the evaluation. (Exhibit 3-3 describes each occupational program in detail.) 


BTC and WCC are located in Bellingham, which is 90 miles north of Seattle. Bellingham has a 
population of approximately 80,000 and is the largest city in Whatcom County, which extends 
north to the U.S.-Canadian border. The median household income in Bellingham in 2015 was 
$43,536, lower than the median for Washington ($61,062) and the United States ($53,889). The 
2015 poverty rate for the city was 23 percent.*? Everett Community College is located about 30 
miles north of Seattle, in Snohomish County. Everett has a population of approximately 
106,000. As in Bellingham, the median income in Everett ($49,578) is lower than in both 
Washington and the United States. Everett’s 2015 poverty rate was 18 percent.*? 


Overall, the local economies in both Bellingham and Everett improved during the years the 
study operated. When BTC began random assignment in November 2011, the unemployment 


31 For additional information about the program see Glosser et al. (2014). 
32_U.S. Census Bureau, 2011-2015 American Community Survey 5-Year Estimates. 


33 U.S. Census Bureau, 2011-2015 American Community Survey 5-Year Estimates. 
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rate in Bellingham was 8.2 percent. In September 2014, when random assignment was 
completed, the unemployment rate was 6.0 percent. In Everett, the unemployment rate 
declined from 6.9 percent to 5.4 percent between February 2013 and April 2014, while random 
assignment was ongoing. 


At the outset of the study, program administrators reported high demand for most occupations 
in I-BEST programs included in PACE, particularly for nursing assistants, welders, and 
machinists. However, BTC stopped its I-BEST Electrical program partway through the study 
because program faculty were concerned that demand for electricians in the construction 
industry had decreased (see further detail below). 


Exhibit 3-1: Location of PACE Study Colleges and Programs Included in the I-BEST Study 


BELLINGHAM TECHNICAL COLLEGE 


WHATCOM COMMUNITY COLLEGE 


°e 
s fe 
* Clerical Assistant y * Precision Machining 


+ Welding 


Mount Vernon 


. 
=" Port Angeles 
EVERETT COMMUNITY COLLEGE 


* Nursing Assistant 
+ Weld 


+ Sustainable Office Skills 


y c Bellevue 


Exhibit 3-2 shows that the three colleges vary in size, student age, and student academic goals. 
BTC serves a much smaller student body (5,526 students) than do EvCC and WCC (19,388 and 
11,292, respectively). BTC also serves a slightly older population (median age of 27 compared 
with 23 or 22). The exhibit also shows the SBCTC average. 
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Exhibit 3-2: Characteristics of Students at Three Colleges and Statewide, 2015-16 
Number of Median Sean Seen Enrolled in 
College Students Age of Skill ss ay Transf Basic Skills 
Enrolled Student peace. oases Courses (%) 
Courses (%) Courses (%) 
Bellingham Technical College 5,526 27 93 2 4 
Everett Community College 19,388 23 40 48 11 
Whatcom Community College 11,292 22 37 56 5 
All SBCTC Colleges 380,918 26 40 41 11 


SOURCE: https://www.sbctc.edu/about/facts-publications/field-guide-2017.aspx. 


At BTC, a much higher percentage students are enrolled in occupational training compared with 
the two other colleges. As shown on Exhibit 3-2, more than 90 percent of BTC students are 
enrolled in occupational skills training, including both short-term certificate and two-year 
degree programs. This is more than double the percentage of students at WCC and EvCC who 
are enrolled in similar programs. At EvCC and WCC, a much higher percentage of students (48 
and 56 percent, respectively) pursue academic or transfer programs—that is, a course of study 
designed to easily transfer to a four-year institution. Finally, the share of students enrolled in 
basic skills classes was low at all three campuses, but less so at EvCC (11 percent) than at BTC (4 
percent) or WCC (5 percent). 


3.1.2. Control Group Environment 


A critical contextual factor for understanding the impact of I-BEST is the degree to which 
educational opportunities and supports comparable to I-BEST’s were otherwise available 
locally. Programs have the greatest potential to produce observable impacts (treatment vs. 
control differences) when the services they offer are clearly distinguishable from those already 
available in the community. The nature of those other services also affects program completers’ 
ability to build on initial training successes. 


PACE control group members had access to educational and occupational supports through 
several means. These included alternative training programs in their communities, non-I-BEST 
courses at the three colleges in the study, and other supports in their communities. Exhibit 3-3 
summarizes the difference in services available to the study’s control group and treatment 
group members, and then each component is discussed below. 
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Exhibit 3-3: Comparison of Career Pathways Components Available to PACE Control Group 
and Treatment Group Members 


Employment e 
Services 


colleges’ advising centers 

College tutoring services 

Financial aid based on eligibility and 
availability (e.g., Pell grants, 
Washington State Opportunity Grants) 


Standard employment services from the 
three colleges’ career services offices 
Job search supports from other 
community providers (e.g., Goodwill 
Industries, American Job Centers) 


Career Standard Community Offerings I-BEST Components 
Pathway Available to Available to 
Component Treatment and Control Groups Treatment Group Only 
Assessment e Accuplacer CASAS 
Instruction e ABE or ESL classes at the three Access to college-level classes and concurrent support 
colleges or other community locations for basic skills needs 
e Non-I-BEST classes at the three Team-taught college-level academic and vocational 
colleges if students met entry classes 
requirements Contextualized basic skills instruction 
e |-BEST classes at other community and Cohort-based learning 
technical colleges Instructor-led study group, open lab, tutoring 
Supports e Academic advising through the three Dedicated I-BEST navigator at each college supported by 


funding from the Open Society Foundations 
Fill-the-gap financial support for tuition and course 
materials 

Funding for support services (e.g., uniforms, 
transportation, licensure testing) 

Career counseling from I-BEST navigator 

Clinical placements (for Nursing Assistant students) 
Internships (for Sustainable Office Skills students) 


SOURCE: Program documents and site visits. 


Assessment. The three colleges assessed students using CASAS prior to random assignment. 


This assessment identified students whose basic skills might not be high enough to otherwise 


qualify them for college-level classes. Students seeking to enroll directly in college-level classes 


(not remedial and not part of I-BEST) take the Accuplacer placement test and need to score 


high enough to qualify for admission. Thus, the colleges use the CASAS to assess basic reading, 


math, and English language skills and the Accuplacer to assess college placement. 


Instruction. Treatment group members could access I-BEST programs and the associated 


occupational and basic skills instruction, including team teaching. Control group members could 


not access I-BEST programs and courses at the three colleges; however, they could access other 


education and training opportunities available to them, including: 


e Non-I-BEST courses. Students in the control group could enroll in non-I-BEST courses at 


the three study colleges, as well as at other colleges in the area if students were eligible. 


For example, both BTC and EvCC offered non-I-BEST versions of Nursing Assistant and 


Welding. Control group students could follow the more typical pathway—taking ABE, 


ESL, or developmental education classes until their skills/placement scores were 


sufficient to qualify them for entry into college-level coursework. 


e |-BEST programs in other local colleges. All three colleges are near at least one other 


community or technical college that also offered one or more I-BEST programs. As a 


Abt Associates 
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condition of participating in the study, control group members were barred from 
enrolling for a designated embargo period in the I-BEST program at the college where 
they applied.** While control group members could enroll in |-BEST programs at other 
colleges that were not a part of the PACE study, SBCTC administrative data show that no 
control group members enrolled in I-BEST courses at other colleges. 


Supports. Treatment group members had access to a dedicated navigator who could provide a 
range of supports throughout students’ enrollment in the I-BEST program. These students also 
could access I-BEST’s fill-the-gap funding and additional funding for supportive services 
available through I-BEST. Both treatment and control group members could access general 
college advising, tutoring, and financial aid services that were available to all students at the 
colleges. Both treatment and control group members could potentially access financial support 
through Pell grants, Washington State’s Opportunity Grants, its Basic Food Employment and 
Training (BFET) program, veteran’s benefits, and Temporary Assistance for Needy Families 
(TANF), depending on eligibility. 


Employment Services. Both treatment and control group members could access their college’s 
employment and job placement services designed to help program completers find jobs. 
Whether they enrolled in college classes or not, they also could access other employment 
assistance in the community, such as the job search and job readiness services at local 
American Job Centers. In addition to these generic services, treatment group members could 
receive job placement and advising from the I-BEST navigator. Several of the I-BEST programs in 
the study also offered clinical placements (e.g., Nursing Assistant) or internship components 
(Sustainable Office Skills) intended to provide relevant work experience as part of the I-BEST 
program. 


In summary, there were alternative options for control group students seeking basic skills, 
academic, and occupational training. They could enroll in |-BEST programs at other schools. Or 
they could enroll in non-I-BEST offerings at their original college or at other public or private 
training providers in the area if they qualified. However, placement elsewhere was subject to 
availability of slots and meeting eligibility requirements. This, as well as the lack of I-BEST’s 
dedicated advising support and fill-the-gap funding, added up to a clear difference in available 
services between the control and treatment groups. 


3.2. I-BEST Program Design 


SBCTC set the general parameters of the I-BEST program, but colleges had flexibility to design 
and structure their I-BEST implementation, most notably how they identified students to enroll 


34 Applicants were informed at intake that if randomly assigned to the control group, they would not be allowed 


to enroll in |-BEST programs at that college for 36 months from the date they entered the study. 
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in the program and the industry focus and structure of the I-BEST programs. For each of the 
colleges in the PACE evaluation, this section describes their goals, recruitment process, and the 
structure and courses for their I-BEST programs. 


3.2.1. Program Goals 


As discussed, the SBCTC designed I-BEST to provide the first step on an articulated career 
pathway in a range of occupations, particularly to students whose skill levels were too low to 
enroll in occupational training otherwise. However, the specific focus and content of the three 
colleges’ I-BEST programs varied, partly reflecting their institutional type: 


e Bellingham Technical College seeks to provide career and technical education geared 
toward developing specific occupational skills. However, the college designed its I|-BEST 
programs so that these short-term credentials were a step on a path toward more 
advanced credentials or associate’s degrees. 


e Everett Community College started its |-BEST program to serve its ESL population. 
Initially it focused on helping ESL students obtain short-term occupational credentials 
for employment in high-demand occupations, and then it expanded to serve the 
broader basic skills population. Although employment can be a shorter-term goal for 
these students, some could choose to continue at EvCC for more advanced certificates 
or associate’s degrees or choose to transfer to a four-year college. 


e Whatcom Community College designed its |-BEST program to serve students with 
specific occupational interests but whose low level of basic skills either made them 
ineligible for admission into their college-level program of interest or suggested they 
would need additional basic skills instruction to succeed in the program. College 
leadership reported that the program was primarily a step toward more advanced 
college certificate and degree programs. This assumption that students would continue 
their education is a key distinction of WCC’s I-BEST program versus the other two 
colleges’. 


3.2.2. Recruitment 


To meet study enrollment goals, participation in the PACE evaluation required the colleges to 
recruit twice as many students for their I-BEST programs, with the goal of recruiting at least 
1,000 study members across the three colleges. Each college was responsible for recruiting 
potential I-BEST participants and developed a recruitment plan that was based on historical 
demand for its I-BEST courses and estimates of the number of students currently in its ABE and 
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ESL programs.*° At the outset of the study, all three of the colleges anticipated that they would 
identify and recruit study participants primarily from these existing student pools. Specifically: 


e Bellingham Technical College expected to recruit students primarily from its ABE 
population. The occupational skills programs that BTC included in its |-BEST program, 
including welding, electrical, and health care occupations, had traditionally been in 
demand by basic skills students at the college. 


e Everett Community College administrators expected to recruit a large share of potential 
I-BEST students from its ABE and ESL programs. In addition, EvCC provides education 
and training services to TANF recipients through WorkFirst (the state’s TANF program). 
Administrators considered the short-term certifications available through I-BEST as a 
good fit for this population, given limitations on the length of training allowed for 
recipients of WorkFirst cash assistance.*° The college also expected to recruit students 
new to EvCC through outreach to local community-based organizations and high 
schools. 


e Whatcom Community College anticipated recruiting most potential Clerical Assistant 
program participants from students currently enrolled in its basic skills classes, including 
ESL. Staff also expected that some students enrolled in developmental education classes 
would benefit from the added support of I-BEST, and planned to recruit from these 
classes, too. 


As discussed in Chapter 4, recruitment proved challenging as the evaluation progressed. As a 
result, each college expanded its recruitment effort to include a larger than anticipated 
outreach to the broader community beyond the existing ESL and ABE classes, as well as 
additional efforts to identify low-skill students who qualified for occupational training but who 
could still potentially benefit from the I-BEST approach. 


3.2.3. I-BEST Program Course Structure, Services, and Staffing 


Exhibit 3-4 summarizes the I-BEST programs included in the PACE evaluation in each of the 
three colleges. It provides the occupational focus, the length of the program, specific courses 
required in each occupation area, credentials that could be attained, and how the program is 
designed to place participants on a pathway toward additional credentials. Each is discussed 
below. 


Occupational focus and program length. As shown, BTC offered the I-BEST programs 
Automotive, Electrical, Nursing Assistant Certified, Precision Machining, and Welding; EvCC 


35 Each college received funding through a subcontract with Abt Associates to support a recruiter. 


36 TANF recipients have a 12-month lifetime limit on vocational education that counts as a core activity, meaning 
that it counts toward a TANF program’s work participation rate. 
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offered Nursing Assistant Certified, Sustainable Office Skills, and Welding; and WCC offered 
Clerical Assistant. The I-BEST programs ranged in length from one quarter to three quarters. 
EvCC included a “pre-I-BEST” quarter focused on basic skills improvements as part of two of its 
three programs. 


Team teaching. All programs included courses that were team taught (these courses are 
indicated in italics in Exhibit 3-4). The team teaching model paired a basic skills instructor and 
an occupational instructor in the delivery of occupational training. 


Basic skills support classes. In addition to the basic skills instruction delivered via team 
teaching, I-BEST programs also often included support classes, taught by basic skills instructors, 
to provide students with the necessary academic foundation to succeed in occupational 
training. Support classes are an opportunity for basic skills instructors to clarify concepts 
presented in the training that might be confusing to students; to address basic skills such as 
reading or math that are required to succeed in training; and to help students prepare for 
upcoming tests, quizzes, and assignments. These basic skills classes are meant to complement 
the occupational courses. 


Credits. As designed by SBCTC, the I-BEST program focuses on the attainment of workforce 
credits that lead to workforce credentials, usually within one to two quarters of programming. 
If they desire, students can progress from there to additional education and training, including 
at the college level, to receive additional workforce credits and credentials. They also can earn 
academic credits and credentials, consisting of credit-bearing courses and credentials that are 
transferrable to a four-year college. Basic skills or developmental courses, including I-BEST 
support classes, ABE, and ESL classes, do not count toward credential attainment. 


Credentials. A range of credentials are available from the I-BEST program, depending on the 
college and occupational field. These include college-awarded credentials that appear ona 
student’s transcript; certificates of completion that indicate the student has completed a series 
of courses but do not appear on a transcript; and state- or industry-recognized credentials that 
require passing an exam administered by a public agency or licensing body. As discussed further 
in Chapter 5, SBCTC awards the following types of credentials: workforce completion (a 
certificate requiring less than 20 credits); workforce award (a certificate requiring 20 or more 
credits but that are not transferable to a four-year college); and associate’s degrees that 
require two years of coursework and are transferrable to a four-year college. Completion of the 
I-BEST courses generally culminated in a workforce award from the college, although further 
state licensing might be required to practice in the field (e.g., nursing or welding). 
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Exhibit 3-4: I-BEST Programs, Courses, Credits and Certifications, and Next Steps, by College 
I-BEST Program / 
Length 
Bellingham Technical College 
Automotive 
(1 quarter) 


Examples of Next Step(s) on 


Courses (team taught are in italics) Ealicshonal Para 


Credits and Certification (if applicable) 


23 Workforce Credits 
Vehicle Service Technician Certificate 


Transportation Services and Systems 
Occupational Math 

Basic Academic Skills 

Trade Safety 

Direct Current (DC) Circuits (including lab) 
Electrical Drawings and Blueprints 
Applied Mechanics 

Occupational Math 

Basic Academic Skills—Math 

Basic Academic Skills—Electrical 


Nursing Assistant Essentials 
Basic Academic Skills 

clinical placement 

one-day courses in CPR and HIV/AIDS 
Machine Lab Safety 

Intro to Measuring and Inspection 
Intro to Machining 

Intro and Advanced Manual Lathe 
Occupational Math 

Applied English 

Machine Fundamentals 

Precision Grinding 

Blueprint Reading 

Intro to Manual Mill 


General Automotive Repair Certificate 
Associate's in Applied Science 
Degree in Automotive Technology 
Electrician Construction Certificate 
Associate's in Applied Science 
Degree 


Electrical 25 Workforce Credits 


(1 quarter) 


12.5 Workforce Credits 
Nursing Assistant Certification (requires passing 
state exam) 


Nursing Assistant Certified 
(1 quarter) 


Phlebotomy 
Medical Coding 
Dental Assisting 


45 credits 
Principals of Precision Machining Certificate 


Precision Machining 
(2 quarters) 


Associate's in Applied Science 
Degree 


Welding 
(2 quarters) 
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Welding Safety | and II 
Shielded Metal Arc Welding 
Thermal Cutting 


e Gas Metal Arc Welding 


Occupational Math 
Applied English 


Career Opportunities for Welders and 


College Success Foundations 


Basic Academic Skills (one per quarter) 


30 Workforce Credits 

Washington Association of Building Officials 
(WABO) Certification (requires passing 
certification exams) 
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I-BEST Program | 
Length 


Courses (team taught are in italics) 


Credits and Certification (if applicable) 


Examples of Next Step(s) on 
Educational Pathway 


Everett Community College 
Nursing Assistant Certified 

(1 quarter, plus ESL/ABE pre- 
program quarter) 


Sustainable Office Skills 
(2 quarters) 


Welding 
(1 quarter, plus ESL/ABE pre- 
program quarter) 


Whatcom Community College 
Clerical Assistant 
(2-3 quarters) 


Pre-I-BEST course 
e Nursing 101 
e |-BEST Basic Skills 


Beginning Keyboarding 

Computer Literacy 

Sustainable Office 

Job Readiness 

Cooperative Work Experience 

Pre-I-BEST course 

Sustainable Industrial Standards for Welding 
Advanced Arc 

Gas Metal Arc/Flux Cord Arc Welding 
|-BEST Basic Skills course 


Introduction to Business Computing 
Office Procedures 

Introduction to Accounting 

Customer Service for Professionalism 
mediated labs in Accounting, Business 
Computing, and Office Administration@ 
e Basic Academic Skills (one per quarter) 


18 Workforce Credits (no credits for pre- 
program) 

Nursing Assistant Certification (requires passing 
state exam) 

19 Workforce Credits 

Sustainable Office Certificate 


16 Workforce Credits (no credits for pre- 
program) 

WABO Certification (requires passing 
certification exams) 


28 Workforce Credits 
Clerical Assistant Certificate of Proficiency 


Phlebotomy Technician Certificate 
Associate's in Applied Science 
Degree in Medical Assisting 


Legal Office Support Certificate 
Medical Administrative Certificate 
Associate's in Technical Arts Degree 


Advanced Tungsten Inert Gas 
Welding 

Aerospace Fabrication & Welding 
Certificate 

Associate’s in Technical Arts Degree, 
Welding 


Accounting Certificate 

Hospitality and Tourism Business 
Management Certificate 
Associate’s in Science Degree in 
Business Administration 


2 Mediated labs are self-directed computer lab courses that allow students to choose from an array of courses in a lab setting with an instructor and teaching assistants, who are available to provide 
support and review work. The class is semi-structured, with quiz and test dates set by the instructor. 
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Subsequent training and credentials. Exhibit 3-4 also shows potential subsequent credential 
options for each of the I-BEST programs in the PACE evaluation. Each of the colleges defined a 
longer-term educational pathway following the I-BEST program that mapped to available 
employment options. Students did not necessarily need to complete all the I-BEST courses ina 
program to move to additional education and training options; in some cases, specific courses 
rather than the completion of the entire I-BEST program were needed to progress to additional 
coursework at the college. 


Exhibit 3-5 below provides two examples of I-BEST educational pathways—the Welding 
program at EvCC and the Clerical Assistant program at WCC. As shown, they provided multiple 
opportunities for occupational training and credential attainment built on the skills students 
acquired during the I-BEST program. 


e EvCC’s Welding program emphasized specialized industry certifications related to 
various welding techniques. As students accumulated these skills, they progressed first 
toward a four-quarter program certificate, and then an Associate Degree in Technical 
Arts. 


e WCC’s Clerical Assistant program provided academic training that could be applied to an 
array of workforce and academic degrees, including Accounting, Business, and 
Entrepreneurship. Students could also earn shorter-term certificates in more specialized 
vocational fields such as Retail Management and Hospitality and Tourism. 


“Fill-the-gap” financial assistance. The colleges expected that most students would receive the 
needed financial support through Pell grants, TANF funds, or other sources such as Washington 
State Opportunity Grants. However, in the event a treatment group member did not qualify for 
aid or tuition costs were not fully covered, the colleges participating in PACE could use OSF 
funding to fill this gap. Thus, fill-the-gap assistance ensured that all treatment group members 
would pay no tuition. 


Dedicated advising. The colleges used foundation funds to add dedicated advisors 
(“navigators”) for |-BEST treatment group members. The advisor was intended to be a regular 
presence in I-BEST classes and available to assist students through all phases of the program, 
including during enrollment, while they were attending I-BEST courses, and with career 
planning. 
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Exhibit 3-5: Examples of I-BEST Educational Pathways 


EvCC WELDING TRAINING 


ASSOCIATE IN TECHNICAL ARTS DEGREE IN WELDING: 
2 YEARS, 90 CREDITS. 


Program Certificate: 4 quarters, 43 credits. 
Many of the same courses as the ATA degree. 


PXe}>jugte) V\etel aul alev vile), ra’ 7 tial [cuge), Wt -lelei/ vile). 
fo) Sm -i8)/Be}] (che) a aleay \e-aa) /\-lep mei gal ater vale), al. 


* SMAW Stick Electrode - Unlimited Plate 
* FCAW Flux Cored Arc Welding - Unlimited Plate 
* FCAW Shop 

* GMAW (Hard Wire MIG) Field and Shop 
* SMAW (Light Gauge) 

* GMAW (Light Gauge) 

* GTAW (TIG) - Unlimited 

* FCAW (Light Gauge) 

* GTAW (Light Gauge) 

* SMAW (Pipe) 

* FCAW (Pipe) 


a» 


I- BEST CERTIFICATE: ENTRY-LEVEL WELDER CERTIFICATE. 
2 QUARTERS, 16 CREDITS. COURSES (ITALICIZED COURSES 
ARE TEAM TAUGHT): 


* Sustainable Industrial Standards 

+ Advanced Arc 

* Gas Metal Arc/Flux Cord Arc Welding 
* ABE and ESL Basic Skills Support 

* Welding Workshop 

* OSHA Safety (for welder certificate) 


» 


: 1 quarter, 12 credits. 
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WCC CLERICAL ASSISTANT TRAINING 


ASSOCIATE OF SCIENCE DEGREES: 2 YEARS, 90-95 CREDITS. 


* Accounting A.S. 

+ General Business A.S. 

* Entrepreneurship A.S. 

* Hospitality and Management Business Management A.S. 


Program Certificate: 4 quarters, 43 credits. 
Many of the same courses as the ATA degree. 


CERTIFICATES IN THE SOCIAL SCIENCE AND BUSINESS 
DIVISION: VARIES (E.G., RETAIL MANAGEMENT IS 46 CREDITS). 
COURSEWORK CAN BE APPLIED TO CERTIFICATIONS IN 

THE FOLLOWING, AS WELL AS PROGRAMS FURTHER UP 

IN THE PATHWAY: 


* Accounting 
* Retail Management 
* Hospitality & Tourism Business Management 


a» 


COURSEWORK CAN ALSO BE APPLIED TO CERTIFICATIONS 
OUTSIDE OF THE BUSINESS DIVISION, SUCH AS: 


* Medical Assisting 


I- BEST CERTIFICATE: CLERICAL ASSISTANT CERTIFICATE OF 
PROFICIENCY. 2 QUARTERS, 28 CREDITS. COURSES (ITALI- 
CIZED COURSES ARE TEAM TAUGHT): 


* Introduction to Business Computing 
* Office Procedures 

* Introduction to Accounting 

* Customer Service for Professionalism 


* mediated labs in Accounting, Business Computing, 
and Office Administration 


* Basic Academic Skills (one per quarter) 
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4. Implementation Study Findings 


Prior chapters described the structure of and services planned by the I-BEST program in the 
three colleges, as well as the contextual factors that could facilitate or impede program 
implementation and outcomes for participants. As described in this chapter, the research team 
examined the extent to which the I-BEST program operated as planned while it was part of the 
PACE study, including efforts to expand and diversify existing recruitment methods in order to 
scale up for the PACE evaluation. 


This chapter describes the implementation findings, based on site visits conducted at two 
points in time. In the second section, it describes patterns in how treatment group participants 
experienced the I-BEST program in the colleges, based on SBCTC program records. It concludes 
by comparing education and training and service receipt for the treatment group versus the 
control group, based on responses to the 18-month study participant follow-up survey. 


4.1. Implementation Findings 


e Recruitment of students for I-BEST was challenging due to lower than expected demand for the 
programs among existing ABE and ESL students. In response, the colleges expanded outreach, but 
still missed their recruitment goal. 


As discussed in Chapter 3, the colleges in the study expected a high proportion of I-BEST 
enrollments would come from current basic skills students, primarily those in ESL and ABE 
courses. Colleges administrators anticipated a high demand among this population for 
occupational training, and several of the I-BEST programs included in the study (e.g., Welding, 
Electrical) had waiting lists in the quarters leading up to random assignment. 


Despite these expectations, all three colleges struggled, recruiting only 63 percent of the total 
sample goal of 1,000 (or 632 students).*” To address recruitment shortfalls, colleges expanded 
their outreach to the general community and students who had qualified college-level 
occupational training based on the entrance exam but still had low basic skills. In particular, the 
colleges adopted two new strategies: 


e Targeting individuals not currently enrolled in the colleges. External recruitment efforts 
included outreach to community-based organizations and employers, and 
advertisements in local media. Additionally, the colleges marketed I-BEST to individuals 
enrolling through the TANF WorkFirst program. 


37 Lower than expected enrollment in I|-BEST may also have been related to a broader decline in enrollment in 
community and technical colleges in Washington. Between 2011 and 2013, enrollment at two-year institutions 
in the state declined 5.9 percent (WSAC 2015). 
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e Targeting students who were eligible to enroll in non-I-BEST college classes based on 
their Accuplacer scores but whose CASAS scores placed them in the eligible range for 
I-BEST. At BTC, administrators reported that many of the students entering the standard 
occupational training courses had basic skills levels, especially in math, that made it hard 
for them to succeed without extra basic skills supports. In the final year of study 
enrollment (2014), BTC required all applicants to non-I-BEST versions of programs (e.g., 
welding) whose Accuplacer scores were on the low end of the eligibility range to also 
take the CASAS to determine basic skills levels and I-BEST eligibility. The theory was that 
students with low but acceptable Accuplacer scores had basic skills levels that made 
them eligible for |-BEST. These students were offered the opportunity to enroll in the 
study. College leadership reported that they saw this both as a way to potentially 
expand the recruitment pool as well as a way to augment supports for students who, 
though they qualified for college-level classes, could benefit from extra instructional 
support. 


e Ultimately, recruitment struggles led to smaller class sizes and blended classes containing both I- 
BEST and non-!-BEST students. 


Each college expected to enroll sufficient study participants to fill its I-BEST courses. However, 
lower than expected study enrollment often resulted in the colleges not being able to fill all 
available slots in the I-BEST courses, in part because the study required that half of eligible 
applicants be randomly assigned to the control group. Often, this meant smaller class sizes. As a 
result, the colleges filled the extra spaces in the I-BEST classes with students who technically 
were not eligible for |-BEST due to their higher skill levels. In these blended classes, the basic 
skills instructor focused his or her attention on the I-BEST students but also provided some 
support to the non-I-BEST students. *° 


e |-BEST served students with low education and basic skills levels, including 31 percent who had 
less than a high school diploma or equivalent. 


The I-BEST program seeks to increase access to college-level classes for students with low basic 
skills. As discussed in Chapter 2, the high proportion of sample members who had less than a 
high school diploma or equivalent reflects the low basic skills level of the population served. 
More than two-thirds of the sample reported never having attended college, and less than 10 
percent had an associate’s degree or higher. This is due to eligibility criteria that targeted 
students with lower skills levels. 


38 Staff at each of the colleges ensured that the non-I-BEST students in these classes were not control group 
members. 
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e Though there was variation across the colleges, the three colleges implemented I-BEST team 
teaching as planned, combining basic skills and occupational technical content. 


A central component of the I-BEST model is team teaching, which pairs an occupational 
instructor and a basic skills instructor in the same course. SBCTC required team teaching, but 
allowed flexibility in how instructors approached their teaching, including how they defined 
their roles within the classroom, planned their time together, and established the rapport to 
work collaboratively. 


Though their role varied across the programs, the basic skills instructors were a key element 
across all |-BEST programs included in the PACE study. As required, the basic skills instructors 
were present for 50 percent of class time for occupational courses; in some cases, they were 
present for almost all of the class time as well as lab time in occupational courses. 


e Shared course time and team teaching were standard across all I-BEST programs, but instructors 
had flexibility in how they approached their teaching. 


Based on discussions with instructors and classroom observations, the research team identified 
three approaches to team teaching: 


e Basic skills instructor as an active student—The basic skills instructor sits in class with 
the students and stops the occupational instructor to ask clarifying questions or fora 
concept to be further explained if students seem confused. This approach was used 
often in occupational courses that were primarily lecture based and highly technical, 
such as Welding and Machining. The approach also was easier to implement for teams 
who had not worked together before or had limited planning time. The box 
Observations from a Precision Machining Class below provides an example of this 
approach to team teaching. 


e Basic skills instructor delivers part of the content during a designated part of the 
occupational classes—For example, a basic skills instructor might start each class with a 
discussion of study skills (e.g., test-taking) before turning the class over to the 
occupational instructor to present technical information. The box Observations from an 
Office Administration Class below contains an example. Staff reported that this 
approach was easier to implement for instructor teams new to each other. 


e Basic skills and occupational instructors jointly deliver class content (co-teaching)—From 
the students’ perspective, delivery is seamless; it may not be apparent which instructor 
is responsible for which type of content. Staff reported this approach was more suitable 
to I-BEST programs where the occupational focus was not very technical or highly 
specialized, such as office skills versus welding. Staff also reported this approach was 
used more often when the team had experience teaching together. The box 
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Observations from a Sustainable Office Skills Class below describes an example of this 
approach. 


Observations from a Precision Machining Course—Active Role for Basic Skills Instructor in 
Traditional Lecture Class at BTC 


The second course in a two-quarter introductory machining sequence was a math-intensive course that 
emphasized “shop essentials.” The objective of the course was to familiarize students with measurement tools 
that they would be using in the shop. The class was primarily lecture based and led by the occupational 
instructor. The instructor explained the use of different tools and parts, the application of geometry concepts to 
the machining tools, and demonstrated measurement techniques using the tools. 


The basic skills instructor frequently summarized and reiterated key points and clarified concepts that seemed 
to confuse students. The basic skills instructor regularly posed questions to the occupational instructor for 
further clarification, such as differences in certain cutting techniques and when certain techniques should be 
used. This resulted in regular dialogue between instructors. 


Both instructors responded to student questions, answering depending on their expertise. When answering 
questions, the basic skills instructor focused primarily on math concepts and terminology, whereas the 
occupational instructor described tools and measurement strategies with some reference to math concepts. 


Observations from an Office Administration Course—Varied Instructional Techniques at WCC 


Office Procedures was a required course in the Clerical Assistant sequence. It focused on business English, time 
and resource management, telecommunications knowledge and phone etiquette, organizational skills, basic 
financial tasks, administrative skills, oral presentations, business terminology, and conflict resolution in an office 
setting. 


Roughly half the observed class was an interactive large group discussion, with the remainder evenly split 
among lecture, independent work, and small group work. The lecture component began with the basic skills 
instructor giving a brief presentation on test-taking strategies. This was followed by a seminar-style discussion 
led by the occupational instructor. Students had the option to work independently or with a partner ona 
worksheet activity about time management, followed by group work in which small groups of students were 
tasked with developing strategies for effective time management given scenarios that were assigned by the 
instructor. The two instructors were purposeful in drawing out students’ personal experiences and attempting 
to make them relevant to the course content. 


The occupational instructor delivered most of the content, and the basic skills instructor offered additional 
clarification or examples. The two instructors traded off in assisting students or calling on students to speak 
during activities. The content was not overly technical, so the occupational and basic skills instructors both 
spoke about general success strategies, skills, and concepts. 
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Observations from a Sustainable Office Skills Course—Integrated Team Teaching at EvCC 


The Sustainable Office Skills class was structured around the creation of a portfolio for each individual student. 
It included both independent and small group work, as well as some interactive content often based on 
assigned readings. 


For the group work, instructors spent a substantial amount of time orienting students to the task and explaining 
the goal of the project. Students worked in pairs or groups of three to develop a sample question to ask an 
employer to improve sustainability in the workplace (e.g., “Have you considered using a recycling bin?”). After 
working in groups, the instructors ask students to share their ideas, and the basic skills instructor defined terms 
that English language learners may have found difficult to understand. Students worked independently to write 
individual 30-second elevator speeches about themselves in preparation for job interviews. The occupational 
skills and basic skills instructors also role-played as job candidate and potential employer, demonstrating 
interview and job readiness skills. 


Instructors were highly integrated in delivering content, with limited differentiation of the basic skills and 
occupational instructor roles. The two instructors interacted constantly throughout the class, with both 
clarifying concepts and delivering instructions on activities and assignment expectations. The basic skills 
instructor asked clarifying questions both to students and to other instructor, provided feedback to student 
responses, and explained classroom tasks. 


e The use of basic skills support courses varied by college and I-BEST program. 


As discussed in Chapter 3, in addition to team teaching of basic skills and occupational training, 
I-BEST programs can also provide separate non-credit basic skills courses designed to 
complement the occupational courses. In these support courses, the basic skills instructors 
review occupational material with the students, help them with their homework, and help 
them prepare for tests and quizzes. 


Both EvCC and WCC required I-BEST students to attend support courses concurrent with 
occupational courses for all their programs. College staff viewed the support courses as way to 
build basic skills concurrent with occupational training. In addition, for students who did not 
have a high school diploma or equivalent and were considering pursuing a GED, these courses 
could help them prepare. 


At the outset of the study, BTC also included support courses for all its |-BEST programs. 
However, in the final year of study enrollment, the college administrators eliminated these 
courses, reporting that instructors found greater value in increasing team teaching time to 
reinforce key course content and to address basic skills gaps. For example, administrators were 
concerned that BTC’s Nursing Assistant students were not being adequately prepared for the 
state certification test. To address this concern, the basic skills instructors spent more time in 
the occupational training classes to help ensure comprehension of core course principles and to 
support increased pass rates on the state exam. 
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e College staff reported that instructors with experience in ABE and ESL programs were well suited 
to serve as basic skills instructors for the team teaching in the I-BEST program. 


Occupational instructors must have content knowledge, and many come from the relevant 
industry, but basic skills instructors can and do come from a variety of backgrounds. College 
staff reported positive experiences in I-BEST with basic skills instructors who could align their 
instruction with the basic skills deficit of -BEST students and that many of these instructors had 
prior experience teaching ABE and/or ESL classes. 


In the interviews conducted for this study, I-BEST basic skills instructors emphasized how the 
learning styles and required supports of the I-BEST students differed from non-l-BEST students, 
in large part due to the former’s lower basic skills or more limited educational experiences. For 
example, staff consistently reported the importance of repeating concepts because I-BEST 
students were reticent to ask clarifying questions. The I-BEST instructors also noted that the 
occupational instructors with less prior experience teaching I-BEST students often 
overestimated what students knew and understood of core concepts, such as math skills for 
Welding students or basic anatomy for Nursing Assistant students. 


e Experienced teaching teams had well-developed roles and teaching strategies for their courses. 


As discussed, the I-BEST program emphasizes the importance of a collaborative teaching model 
in which instructors work together to develop and deliver curriculum, but is less prescriptive in 
defining their specific roles. The research team observed how instructional roles varied across 
teaching teams and across the programs in the three colleges. In more established teaching 
teams (i.e., instructors who worked together in at least one previous course), basic skills 
instructors took a more active role in the occupational courses. These instructors reported that 
their earlier work with the occupational instructor gave them a greater understanding of course 
content, and thus they felt more comfortable taking an active role. Occupational instructors in 
these teams reported that their earlier work with the basic skills instructor gave them a greater 
appreciation for the needs of I-BEST students and for adult education techniques. 


Both EvCC and WCC had established teaching teams that worked with each other for multiple 
quarters. These teams noted that they had a clearer shared understanding regarding their roles 
within the classroom. In some teams, instructors had equal responsibility for delivering course 
content. Examples included instructors splitting class time, with each addressing a particular 
topic during a given lecture, or classes where there was less emphasis on lecture and more on 
jointly facilitating small group work with students. Other established teaching teams adhered to 
a more traditional division of labor, with the basic skills instructor taking a secondary role. 


Instructors also reported that the earlier team experience was beneficial because the basic 
skills instructor typically then had command of the occupational material, including asking 
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relevant clarifying questions, providing examples that were more likely to resonate with 
students, or providing advice about preparation for upcoming assignments. Additionally, staff 
reported that the accumulated expertise of these basic skills instructors was helpful when they 
attended courses with lab components (e.g., Welding, Nursing Assistant), as they were better 
equipped to provide students with guidance on the technical aspects of the applied 
coursework. 


e Reliance on adjunct instructors resulted in limited planning time for teaching teams. 


As discussed in Chapter 1, colleges are reimbursed at 1.75 times the regular rate for a FTE 
student to help cover the costs of implementing I-BEST, including instructor preparation. In 
interviews conducted for the PACE study, college staff reported that teaching teams did not 
have much, if any, preparation time. This was due in part to the adjunct status of the basic skills 
instructors, who did not have full-time positions with the colleges and who were paid by the 
course. Though these instructors had experience with ABE and ESL students, initially they often 
had no experience with the occupational content area of the I-BEST program to which they 
were assigned. Many of the occupational instructors also were adjunct faculty. This staffing 
approach is common for community and technical colleges across the country, in large of part 
to reduce staffing costs.7? However, the reliance on adjuncts resulted in instability in teaching 
teams over time and reduced opportunities for collaborative course planning and professional 
development. 


College administrators reported that the adjunct staffing model most affected the time 
instructors might meet in advance of the quarter to discuss their team teaching strategies. 
Instructors at all three colleges reported meeting in advance of the start of the quarter, but 
they typically were not paid for this planning time. Staff reported that most teaching teams 
developed the necessary rapport to effectively deliver the I-BEST program, but that newer 
teaching teams and those with limited planning time often continued to develop and finalize 
their approach to team teaching in the early weeks of the quarter. 


e Opportunities for instructor training were limited, which most affected the newer teaching teams. 


SBCTC provided professional development opportunities for I-BEST instructors across the state 
at various points during the study, and instructors at the three colleges reported that they 
attended one or more of these. These opportunities were typically conferences offered 
annually that featured presentations from experienced I-BEST instructors and SBCTC staff. 
Examples of presentations included a demonstration of a team teaching approach to content 


39 A recent study found 68 percent of instructional staff at public two-year colleges were classified as part-time 


(Ginder et al. 2017). 


Abt Associates 4. Implementation Study Findings | pg. 43 


Washington State’s Integrated Basic Education and Skills Training (I-BEST) Program in Three Colleges 
Implementation and Early Impact Report 
PACE 


delivery, a group discussion among instructors of approaches they found to be effective, and 
information provided by the SBCTC of planned changes to I-BEST program offerings. 


Generally, it was the basic skills instructor who attended these professional development 
conferences; it was rare for both members of a teaching team to have attended. This, 
combined with the lack of paid preparation time, meant that newly paired instructors had to 
learn each other’s styles and preferences as the course proceeded. Conversations with 
instructors and classroom observations suggested that all instructors delivered the core 
components of the I-BEST model as intended, but it was this learning curve, especially early in 
the quarter, which made integrating their roles difficult. 


e As designed, the I-BEST programs used additional funding during the PACE evaluation to provide 
“fill-the-gap” financial assistance and student advising through paid “navigators.” 


I-BEST staff reported that their students received a range of financial assistance and supports 
that were not available to other students in the college. This included assistance in completing 
the Free Application for Federal Student Aid (FAFSA). Many I-BEST students were new to college 
and had never filled out the FAFSA or applied for other forms of financial aid. Navigators 
regularly helped new students apply for financial aid for which they were eligible. The I-BEST 
program then provided “fill-the-gap” funding if the financial aid was not sufficient. These funds 
also supported books, tools, or work clothes (e.g., steel-toed boots for welding labs, uniforms 
for nursing assistants during clinical placements), transportation (e.g., bus passes, gas cards), 
and licensing/certification fees. 


The implementation study documented that navigators proactively provided advising to I-BEST 
students that was not available elsewhere on campus. Each navigator reported regularly 
meeting with students to discuss class progress, identify potential barriers to participation, and 
help with career and academic planning. They also occasionally attended I-BEST classes to 
check in with students. Navigators reported that often, by virtue of relationships that they 
developed during recruitment for the program, students came to see them as a regular fixture 
in their campus life. For example, instructors and administrators at BTC characterized the 
navigator as a “one-stop-shop” who served as both a coordinator and a navigator of student 
services on campus for I-BEST treatment group members. Another administrator described the 
navigator role as an “early alert system” to proactively identify potential concerns that would 
inhibit student success. 


e Basic skills instructors provided advising and support, specifically around academic issues. 


In addition to the advising and support role played by the navigators, the basic skills instructors 
at each of the three schools also developed relationships with I-BEST students and provided 
ongoing support, particularly around academic issues. Though each of the three colleges 
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offered tutoring to all students, college administrators reported that I-BEST students often 
were hesitant to engage with these services and were more inclined to seek the assistance of 
the basic skills instructors, whom they already knew. 


Based on the research team’s classroom observations and interviews with instructors, the basic 
skills instructors appeared to actively cultivate these relationships, seeking to build trust with 
students to identify and address their academic needs. Though this support most often 
occurred in the classroom or in a lab session, the basic skills instructors made themselves 
available to students outside of class if they required additional academic assistance. Some of 
the basic skills instructors reported providing tutoring and one-on-one academic support, 
especially in the beginning of the quarter, for those students who appeared to be struggling 
with course material. These instructors also provided more informal ongoing supports to 
students. For example, instructors would regularly remind students about upcoming 
assignments, discuss positive time management habits, or provide examples of approaches to 
studying for upcoming tests or quizzes. 


e The I-BEST program did not offer structured services to students in finding employment, but some 
instructors provided informal employment and job search assistance. 


SBCTC designed the I-BEST program with an explicit focus on occupational training that could 
increase the earnings potential of enrolled students. However, none of the three colleges had 
specific employment supports for I-BEST students. Administrators at one college noted that 
employment services were a weakness of the I-BEST program at the college, whereas another 
administrator indicated that the lack of employment services specific to |-BEST reflected the 
focus on continued training and credentials. 


In part because their colleges’ formal employment services were limited, staff reported that 
basic skills instructors and occupational instructors provided informal, individualized assistance 
to students as requested. For example, some I-BEST instructors maintained ongoing 
relationships with employers and mapped instructional content to skills that employers 
reported were in demand. These were often instructors who had come out of industry and had 
knowledge of the local labor market. I-BEST instructors provided examples of how they 
connected students directly with employers. For example, one Welding instructor kept apprised 
of employers likely to hire students who completed the one-quarter entry-level certificate. The 
instructors provided applications for employers and coached students on the application 
process. 


4.2. Education and Training Participation Patterns 


One of the central objectives of I-BEST is to increase students’ participation in and receipt of 
workforce and other credentials at two-year colleges, as well as for them to move on to 
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additional education and training after I-BEST, as appropriate. For each of the three colleges, 
this section analyzes participation in the I-BEST programs, credential receipt, progression to 
additional training after I-BEST, and length of stay in I-BEST as well as other college courses, for 
students assigned to the study’s treatment group. The analysis, based on program records from 
SBCTC, reports the overall level of participation and credential receipt over a 24-month 
follow-up period. 


e Welding and Nursing Assistant were the most common programs in which students randomly 
assigned to the treatment group enrolled. 


As discussed in Chapter 2, PACE study participants were randomly assigned to the treatment or 
control group for the occupational program in which they were interested. For example, a 
student interested in enrolling in the Welding program at BTC would apply for that specific I- 
BEST program and would be able to enroll if he or she was randomly assigned to the treatment 
group. Exhibit 4-1 shows the distribution of treatment group members across I-BEST programs 
for each of the three colleges and overall. Among those seeking to enroll in |-BEST courses 
across the three colleges, the Nursing Assistant I-BEST programs had the greatest proportion of 
treatment group members 36 percent of the 315 treatment group members (25 percent at BTC 
and 10 percent at EvCC). Another 30 percent of those randomly assigned had applied to I-BEST 
Welding programs (split evenly at BTC and EvCC). One quarter of treatment group members 
were randomly assigned to the I-BEST programs related to office and clerical work: Sustainable 
Office Skills (13 percent at EvCC) and Clerical Assistant (12 percent at WCC). Only a small 
portion of the treatment group sought training as machinists, electricians, and automotive 
workers. The largest proportion of the treatment group (50 percent) was randomly assigned at 
BTC, followed by EvCC (38 percent), and WCC (12 percent). 


Exhibit 4-1: Percentage of Treatment Group Members Enrolled in I-BEST, by College and 


I-BEST Program 
I-BEST Program BIC EvCC Wcc Total 
Automotive 1% 
Clerical Assistant 12% 
Electrical 5% 5% 
Nursing Assistant 25% 36% 
Precision Machining 3% 3% 
Sustainable Office Skills [== 13% 13% 
Welding 15% 15% 30% 
Total 50% 38% 12% 100% 


SOURCE: SBCTC program records. 
NOTES: Sample size is 315 and includes all individuals assigned to the treatment group. Percentages may not sum to total due to rounding. 


e Across the three colleges, close to three-quarters of treatment group members randomly assigned 
to I-BEST programs participated in one. 


Of the 315 treatment group members randomly assigned to an I-BEST program, some did not 
participate in the I-BEST courses. Exhibit 4-2 shows the proportion of the treatment group that 
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participated in an I-BEST program; and for those who participated, the average number of 
credits earned and average length of stay while in I-BEST (in quarters). 


As shown in the second column, 73 percent of treatment group members participated in at 
least one I-BEST course at one of the three colleges (defined as attending at least one team- 
taught I-BEST course in their chosen occupational area). Almost 90 percent of the WCC 
treatment group participated in an I-BEST course, whereas 76 percent of BTC students and 64 
percent of EvCC students did so. Participation rates were above 80 percent for the Automotive, 
Electrical, and Welding programs at BTC, the Sustainable Office Skills at EvCC, and the Clerical 
Assistance program at WCC. The lowest rate of participation (48 percent) was in the I-BEST 
Welding program at EvCC. 


The third column of Exhibit 4-2 shows the number of workforce credits that treatment group 
members earned while in I-BEST. Across the three colleges, the 230 who participated in I-BEST 
(as opposed to the 315 in the full sample of treatment group members) earned an average of 
14 credits. However, as expected given the different credit requirements and length of each 
I-BEST program, the number of credits treatment group members earned varied by program. 
Treatment group members who participated in the two-quarter Precision Machining and 
Welding programs at BTC earned the largest number of credits (43 and 33, respectively), 
whereas those who participated in the one-quarter Nursing Assistant program earned among 
the lowest number (7.1 at BTC and 9.5 at EvCC). 


The final column of the exhibit shows the average number of quarters that treatment group 
members attended in the I-BEST program. This again reflects only the 230 who attended as 
opposed to all 315 treatment group members randomly assigned. By design, I-BEST programs 
are short, and students attended 1.5 quarters on average. Again, the number of quarters 
attending the I-BEST program was longer in those programs with longer requirements (BTC’s 
Precision Machining and Welding, 2.2 and 2.7 quarters, respectively). In most instances, the 
average number of quarters students attended is longer than the planned length, reflecting 
that some students likely did not complete their I-BEST programs in the specified number of 
quarters and thus enroll in |-BEST courses in a subsequent quarter. This is likely a combination 
of students needing more time to complete the course requirements associated with a given I- 
BEST program and students re-enrolling in the same classes a second time after having dropped 
out of the course prior to completion at an earlier date. 
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Exhibit 4-2: Participation Rates in I-BEST Program for Treatment Group Members and Average 
Credits Earned and Quarters Attended by I-BEST Participants, by College and Program, within 
24 Months of Random Assignment 


Bye A Average Number of Average Number of 
Goll eget rogram Numba POniciened in Mie Workforce Credits Earned pier Attended in 
Randomly Assigned, Program While in |-BEST Programe I-BEST Program 
and Planned Length of Se a eo 
I-BEST Program Randomly Assigned Among Those Participating in |-BEST 
Automotive (N=3) 100% 18.3 1.0 
(1 quarter) 
Electrical (N=15) 87% 13.1 1.2 
(1 quarter) 
Nursing Assistant (N=80) 71% 7.1 11 
(1 quarter) 
Precision Machining (N=10) 60% 43.2 2.2 
(2 quarters) 
Welding (N=48) 81% 32.7 2.7 
(2 quarters) 
Subtotal (N=156 76% 18.3 1.7 
Everett Community College 
Nursing Assistant? (N=33) 61% 9.5 1.0 
(1 quarter) 
Sustainable Office Skills (N=42) 86% 11.6 15 
(2 quarters) 
Welding? (N=46) 48% 10.5 0.9 
(1 quarter) 
Subtotal (N=111) 65% 10.7 12 
Clerical Assistant ) (N=38) 89% 9.1 17 
(2-3 quarters) 
Total 73% 14.4 15 


SOURCE: SBCTC records. 

NOTES: Sample size is 315 and includes all individuals in the treatment group for second column, and 230 (all individuals who attended at 
least one team-taught |-BEST course) for the remaining columns. 

4 In the case of the Nursing Assistant and Welding programs at EvCC, the occupational training occurred within a single quarter, but 
students were first required to attend a pre-I-BEST quarter that focused on ABE and ESL training, immediately prior to the start of the I- 
BEST program. The proportion who attended this pre-quarter activity is not shown on the table. Program records show that 67 percent of 
treatment group members enrolled in the Nursing Assistant course at EvCC attended its pre-I-BEST course, whereas 50 percent of Welding 
enrollees did so. 

> This is defined as enrolling in at least one team-taught |-BEST course. 

© This includes a small number of academic credits (0.7) on average across the colleges and programs. As discussed in Chapters 3 and 5, 
basic skills credits and developmental education credits do not count toward credential attainment and are not included in the PACE 
analysis. 


More than one quarter (27 percent) of treatment group members did not participate in an I- 
BEST program. However, based on SBCTC records, approximately half of those who did not 
participate in I-BEST enrolled in at least one other course at one of Washington’s community or 
technical colleges but did not enroll in I-BEST (not shown).*° The PACE 18-month follow-up 
survey also asked treatment group members who did not attend any education or training 


40 SBCTC records show that no treatment group members participated in I-BEST at any college besides the three 
involved in the PACE study. 
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(I-BEST or otherwise) for the most important reasons they did not. The most common reasons 
reported were not having enough time due to work or family, or the belief that they could not 
get enough financial assistance (not shown). 


e  Fifty-eighty (58) percent of treatment group members who enrolled in I-BEST obtained a 
credential, and a large proportion (63 percent) of those who participated in I-BEST attended 
additional education and training. 


A key goal of I-BEST is to provide students with credentials in their area of study. As discussed in 
Chapter 3, each of the colleges awarded a college-issued certificate for the completion of the 
I-BEST program, with some programs (e.g., Nursing Assistant, Welding) also requiring additional 
certification from an outside organization or the state. Exhibit 4-3 reports credentials awarded 
by each college, based on SBCTC program data; values do not reflect any credentials earned 
from outside licensing bodies. 


As shown in the first column on Exhibit 4-3, based on SBCTC data, 58 percent of treatment 
group members who participated in I-BEST received a credential by the end of the 24-month 
follow-up period. Most were workforce credentials (not shown).*? One limitation of these data 
is that they do not indicate the occupation or type of workforce credential obtained, nor 
whether it was associated with participation in |-BEST. However, 60 percent of the credentials 
earned were within the first six months of the follow-up period, (not shown). This suggests that 
many were earned as a result of completing an I-BEST program. 


Credential receipt varied by college and occupational area. More than three-quarters of EvCC 
I-BEST students who enrolled in an I-BEST program (76 percent) obtained a workforce 
credential, compared with 55 percent of BTC students and 29 percent of WCC students. 
Credential receipt rates were 80 percent or above for treatment group members who 
participated in BTC’s Precision Machining and EvCC’s Nursing Assistant and Welding programs. 
EvCC had notably higher credential receipt rates than did comparable programs at the other 
colleges for both Welding (95 percent for EvCC versus 38 percent for BTC) and clerical programs 
(61 percent for EvCC versus 29 percent for WCC). 


41 Of the credentials awarded, 84 percent were workforce credentials (requiring 20 or more credits), 13 percent 


were workforce awards (requiring less than 20 credits), and 1 percent were academic credentials. 
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Exhibit 4-3: Credential Attainment and Post-I-BEST College Enrollment Among Treatment 
Group Members Who Participated in the I-BEST Program, Within 24 Months of Random 
Assignment 


: Average Number 
Obtained aces of Quarters Still Enrolled 
College Program Workforce Courses After Attended, I-BEST at End of the 
Credential -BEST Plus Additional Follow-up Period 
Courses 

Bellingham Technical College 

Automotive (N=3) 33% 67% 5.0 0% 
Electrical (N=13) 31% 77% 43 23% 
Nursing Assistant (N=57) 70% 53% 3.7 28% 
Precision Machining (N=6) 83% 50% 4.0 0% 
Welding (N=39) 38% 72% 5.3 18% 
Subtotal (N=118) 55% 62% 44 22% 
Everett Community College 

Nursing Assistant (N=20) 80% 55% 3.4 20% 
Sustainable Office Skills (N=36) 61% 67% 40 25% 
Welding (N=22) 95% 55% 3.1 9% 
Subtotal (N=78) 76% 60% 3.6 19% 
Whatcom Community College 

Clerical Assistant (N=38) 29% 76% 45 32% 
Total 58% 63% 41 23% 


SOURCE: SBCTC program records. 
NOTES: Sample size is 230 and includes all individuals who attended at least one team-taught I-BEST course. 


The second column of Exhibit 4-3 shows that a high proportion of treatment group members 
who participated in I-BEST programs continued their education beyond I-BEST, with 63 percent 
enrolling in additional college courses following their enrollment in I-BEST. This rate was similar 
at BTC and EvCC (about 60 percent) but higher at WCC (76 percent). Students in the clerical 
programs at both EvCC and WCC participated in additional training at a relatively high rate 
(more than two-thirds did so); whereas the rate was lower for both Nursing Assistant programs 
and BTC’s Precision Machining (about 50 percent). Again, there were differences in the two 
Welding programs, with 72 percent of those participating in additional training in the BTC 
program and 55 percent doing so at EvCC. 


The variation in credential receipt rate, as well as the high rate of progression to additional 
education and training (in some cases greater than receiving a credential through I-BEST), 
results from I-BEST students not completing all of the course requirements but still continuing 
on to higher or perhaps different courses of study. Staff reported some staff and students 
focused on progressing to the next level of training. That is, students may not have completed 
all the requirements to obtain the college-issued credential, but still met the prerequisites for 
proceeding to the next level of training. Staff also reported that some students changed their 
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focus or interests while attending I-BEST, and thus continued without completing an I-BEST 
credential. 


In addition, because of the labor market, some programs de-emphasized obtaining the 
short-term college-issued certificates of |-BEST completion and encouraged progression to 
higher levels of training. For example, EvCC’s Welding program focused on helping students 
attain an industry-recognized credential after the one-quarter program because there were 
entry-level production welding job opportunities (hence its credential receipt rate of 95 
percent). By comparison, BTC’s Welding faculty focused less on near-term certifications, instead 
they believed that students would be most employable in the local labor market after a full year 
of training. As a result, BTC Welding students were more likely to continue in welding without 
necessarily receiving an occupational credential; 72 percent of those who attended the BTC I- 
BEST Welding program progressed to additional education and training. Similarly, WCC staff 
encouraged their Clerical Assistant students to pursue education and training beyond an entry- 
level clerical position (with only 29 percent receiving the I-BEST credential and 76 percent 
attending subsequent education and training). 


Based on SBCTC program records, for the 230 treatment group members who attended an 
I-BEST program, the average length of participation for all education and training programs 
(including I-BEST) over the 24-month follow-up period was 4.1 quarters.” Length of stay was 
more than five quarters for those who had attended BTC’s Automotive and Welding I-BEST 
programs, and lowest (about three quarters) for those who had attended EvCC’s I-BEST Nursing 
Assistant and Welding programs. 


At the end of the 24-month follow-up period, about one-quarter (23 percent) of the treatment 
group that attended I-BEST was still participating in at least one course at an SBCTC college. This 
proportion was lowest at EvCC (19 percent) and highest at WCC (32 percent). The proportion 
still participating was relatively high at both clerical programs, as well as at BTC’s Electrical and 
Nursing Assistant programs. No students who participated in BTC’s Automotive or Precision 
Machining I-BEST programs were still participating at the end of the follow-up period. 


4.3. Impact on Receipt of Services 


The previous section focused on treatment group member experiences, using SBCTC data. This 
section focuses on the degree to which I-BEST increased receipt of education and training, 
advising, and employment services and is based on data from the 18-month survey. An 
implication of the career pathways framework is that any improvements in the main outcomes 


42 This length of participation is truncated because some students were still attending programs at the end of the 


follow-up period. 
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(discussed in Chapter 5) will result primarily from impacts on the treatment group’s experiences 
and services tied to education and training. 


The purpose of this section is to compare levels of education and training receipt between the 
treatment and control groups—that is, the service differential that is expected to lead to 
impacts. (The box below briefly explains how to read impact tables.) Specifically, the following 
section discusses impacts on education or training receipt after random assignment (Exhibit 
4-4) and receipt of advising and employment services (Exhibit 4-5). 


How to Read Impact Tables 


Exhibit 4-4 and Exhibit 4-5, as well as exhibits in Chapter 5, list the outcome measure in the analysis in the left- 
most stub column (Outcome), with the unit of that outcome in parentheses (e.g., “(%)”). 


The next column (Treatment Group) presents the treatment group’s regression-adjusted mean outcome, 
followed in the next column by the control group’s actual mean outcome (Control Group). The regression 
adjustments correct for random variation in baseline covariates between the two groups (and thus differ slightly 
from the raw means). The next column (Difference) lists the impact—that is, the difference between the 
treatment and control group means. The Standard Error column is a measure of uncertainty in the estimated 
impact that reflects both chance variation due to randomization and any measurement error. 
The final column, p-Value, is the probability that observed or larger difference between the treatment and 
control group would occur by chance, even if there was no difference in the characteristics of the two groups. 
There are several common standards for judging statistical significance. In this report, tests are considered 
statistically significant and highlighted in tables if the p-value is less than or equal to .10. Tests with smaller p- 
values are separately flagged: 

*for.10 (ten percent level) 

** for .05 (five percent level) 

*** for .01 (one percent level) 
Outcomes in italics apply to a subset of survey respondents (e.g., those who attended education or training). 
These estimates are not impacts, but unadjusted, non-experimental comparisons. 


e |-BEST had a statistically significant impact on education and training receipt. 


Exhibit 4-4 shows statistically significant impacts on study participants’ receipt of any education 
and training activities based on responses to the 18-month follow-up survey. I-BEST increased 
receipt of training in any subject by 23 percentage points (76 percent of treatment group versus 
53 percent of control, statistically significant at the one percent level). Though the only I-BEST 
healthcare training program was Nursing Assistant (at BTC and EvCC), it was one of the largest, 
and overall it increased receipt of healthcare-related training by 13 percentage points (29 
percent versus 16 percent, statistically significant at the one percent level).*% 


43 These proportions represent the percentage of treatment and control group members who reported on the 
follow-up survey that they participated in an education or training program at the three colleges or elsewhere. 
Though similar, this value differs from the proportion who participated in a program based on administrative 
data. This difference is due to variation in the data source (self-reported measures are subject to recall error). 
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Exhibit 4-4: Education or Training Receipt After Random Assignment 


Outcome Treatment Control Difference Standard p-Value 


Group Group Error 
General Aspects of Education & Training Receipt 


Received education or training since random assignment (%) 


In any subject/field 76.0 53.3 +22.7 ** 4.5 <.001 
In a healthcare occupation 28.9 15.6 +13.3 3.5 <.001 
Since random assignment, ever attended (%) 
Two-year college 70.7 45.9 424.8 ** 4.5 <.001 
Four-year college 5 1.5 +0.0 1.3 .978 
Proprietary school 1.2 0.9 +0.4 0.9 .665 
Adult high school/education 0.9 0.5 +0.4 0.9 675 
Community/nonprofit organization 0.9 17 -0.8 1.2 522 
Other 45 6.9 -2.4 2.0 224 
Time spent at school and work at first place attended (%) 
Full-time school and full-time work 5.0 2.5 42.5 2.1 231 
Full-time school with no or part-time work 42.2 47.4 -5.2 6.1 392 
Part-time school and full-time work 114 8.4 +2.7 3.6 453 
Part-time school with no or part-time work 41.7 41.7 +0.0 6.1 999 
Total 100.0 100.0 
Views of classes at first place attended (%) 
Strongly agrees relevant to life/career 4 57.3 43.8 +13.6 ** 6.1 027 
Used active learning methods most/all of the time ® 39.1 32.7 +6.4 5.8 273 
Perceived strong emphasis on community at first 9.0 14.5 -5.5 41 .180 


place of instruction (% 
Basic Skills Instruction and Tests 


Received basic skills instruction since random assignment (%) 


Academic skills 33.8 26.2 +7.6 * 4.4 .081 

English as a Second Language 18.1 12.5 +5.6 * 3.2 .076 
Took college placement exam (%) 

English 51.7 43.3 +8.4 * 4.7 .076 

Math 48.5 44.4 +4.1 4.7 .387 
Passed college placement exam (%) 

English 39.3 31.3 +7.9 * 4.7 .090 

Math 33.4 30.2 +3.2 4.4 466 
Received life skills instruction since random 26.8 24.3 +25 4.0 538 


assignment (%) 


Sample size (full survey sample) 238 2 Da 


SOURCE: Abt Associates calculations based on data from PACE short-term follow-up survey. 

NOTES: Where not italicized, outcomes apply to the full survey sample, and impact estimates are fully experimental and regression- 
adjusted. Outcomes in italics apply to subset of survey respondents (e.g., those who attend education or training)—for these estimates, 
between-group differences are unadjusted, non-experimental comparisons. 

Statistical significance levels, based on two-tailed t-tests of differences between research groups, are summarized as follows: ***statistically 
significant at the one percent level; ** at the five percent level; * at the ten percent level. 

4 Percentages who either strongly agreed that classes were relevant to career interests or who strongly disagreed that classes did not 
relate to anything else in life. 

> Refers to first place of instruction if went to more than one place. Gives the average percentage who described classes as involving each 
of a series of active learning approaches at least often, or at least most of the time (items used different scales, as discussed in Appendix 
C). 


In addition, treatment group members who did not enroll in the I-BEST program may have enrolled in other 
education and training programs in the community. 
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The program produced an eight percentage point impact on receipt of basic skills instruction 
(34 percent of treatment group members versus 26 percent of control group members, 
statistically significant at the 10 percent level). Given that the I-BEST program allowed members 
of the treatment group to enroll in credit-bearing courses without remediation, this difference 
could be due to respondents in the treatment group interpreting this question to apply to basic 
skills support classes required for many I-BEST certificates, or the basic skills instruction they 
received through the team teaching approach. There was also an eight percentage point impact 
on completion of a college placement exam in English (52 percent of treatment group members 
versus 43 percent of control group members, statistically significant at the 10 percent level). 
There was no statistically significant impact on the percentage reporting haven taken a math 
placement exam. I-BEST significantly increased the likelihood of passing a college placement 
exam in English, but had no effect in math. 


Exhibit 4-4 also shows non-experimental comparisons for the subset of treatment and control 
group survey respondents who reported attending any education and training (see italicized 
rows). Treatment group members attending an education and training program were more 
likely than the control group to agree that the courses at the first place they attended were 
relevant to their life/career (57 percent versus 44 percent, statistically significant at the five 
percent level). 


e  |-BEST had statistically significant impacts on receipt of advising and employment services. 


Exhibit 4-5 shows impacts on advising and employment services for all treatment and control 
group members, regardless of whether they received services or not. Reflecting the dedicated 
advisor that was part of I-BEST, the program increased receipt of career counseling by 14 
percentage points (36 percent of the treatment group members versus 22 percent of control 
group members, statistically significant at the one percent level). There were similar impacts on 
receipt of help arranging for supports for school, work, or family (29 percent of the treatment 
group versus 16 percent of the control group, statistically significant at the one percent level). 
Though there were no official job search assistance services, |-BEST produced an impact on 
receipt of job search or placement services (29 percent of the treatment group versus 19 
percent of the control group, statistically significant at the one percent level), potentially 
reflecting the assistance received from instructors or the advisor. 


Though these impacts are relatively large, only a minority of the treatment group reported 
receiving advising or employment services. Roughly two-thirds of the treatment group did not 
report this support. 
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Exhibit 4-5: Receipt of Various Supports Within 24 Months of Random Assignment 
Treatment Control Standard 
Outcome Group Group Difference Error _p-Value 
Received assistance from any organization since random assignment (%) 
Career counseling 36.1 22.3 +13.8  *** 4.2 .001 
Help arranging supports for school/work/family 28.9 16.4 +12.5 *** 3.7 <.001 
Job search or placement 29.3 19.0 +10.3** 3.9 .008 
Received supports at first place of instruction attended (%) 
Career counseling 
Ever 26.1 27.4 -1.3 5.5 814 
Three or more times 13.3 7.8 +5.5 3.6 129 
Academic advising 
Ever 62.3 57.0 +5.3 6.1 384 
Three or more times 40.2 41.3 -11 6.1 857 
Financial aid advising 
Ever 54.0 48.4 +5.6 6.2 366 
Three or more times 19.0 23.5 -4.4 5.1 384 
Tutoring 
Ever 41.9 37.6 +4.3 6.0 475 
Three or more times 34.3 33.5 +0.8 5.9 894 
Help arranging supports for school/work 
Ever 23.2 24.6 -1.4 5.3 792 
Three or more times 13.4 10.5 +2.9 3.9 461 
Job search/placement assistance 
Ever 25.6 22.3 +3.3 5.2 525 
Three or more times 10.7 9.2 +14 3.5 .687 
Received financial assistance at first place of instruction (%) @ 
Grant/scholarship 80.2 67.9 +123 * 5.4 .023 
Loan 19.0 10.9 +8.1 * 42 055 
Cited financial support as challenge in enrollment or 56.6 58.9 -2.2 47 .636 
persistence (%) > 
Offered opportunities for related work experience as part of training at first place of instruction (%) 
Clinical internship 23.3 +13.2 ** 5.5 017 
Visits to local employer 25.1 19.1 +6.0 5.0 228 
Work-study job 37.9 25.6 +12.3 * 5.6 029 
Apprenticeship 14.9 11.2 +3.7 4.0 360 
Any related work experience (including other) 64.7 49.0 +15.6 * 6.0 .010 
Sample size (full survey sample) 238 225 


SOURCE: Abt Associates calculations based on data from the PACE short-term follow-up survey. 
Where not italicized, outcomes apply to the full survey sample, and impact estimates are fully experimental and regression-adjusted. 

Outcomes in italics apply to subset of survey respondents (e.g., those who attend education or training)—for these estimates, between- 
group differences are unadjusted, non-experimental comparisons. 
Statistical significance levels, based on two-tailed t-tests of differences between research groups, are summarized as follows: ***statistically 
significant at the one percent level; ** at the five percent level; * at the ten percent level. 
@ Reported receiving grant or loan to help cover either tuition/school expenses or living expenses. 


> Cited financial support challenges as a reason for non-enrollment or leaving school or as a difficulty while attending school. 


Exhibit 4-5 also shows non-experimental comparisons for a subset of treatment group members 


who attended any education or training program. Reflecting the fill-the-gap financial assistance 


advising, among students who enrolled in a program, a significantly higher proportion of 
treatment group members reported receiving grants or scholarships (80 percent) than did 


control group members (68 percent). 
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Among those who attended, there were also positive impacts related to work experience as 
part of the training. Treatment group members were more likely than control group members 
to report participating in any work-related experience (65 percent versus 49 percent), as well as 
ina clinical internship (37 percent versus 23 percent) or a work-study job (38 percent versus 26 
percent). 


4.4. Summary of Implementation Findings 


Though there was variation across the three colleges and within programs at each college, each 
college delivered a version of |-BEST that incorporated the key features of the model as 
designed by the SBCTC. This included increased access to college-level classes for students with 
low basic skills or limited English proficiency, team-taught classes providing both occupational 
training and basic skills instruction, and proactive services related to advising and financial 
assistance. There was variation across the occupational programs in how they implemented the 
team teaching and the extent to which additional support classes providing basic skills 
instruction were added to the occupational training. In general, the |-BEST approach was more 
challenging for new staff members who had limited experience with team teaching and/or the 
specific course content; they experienced a learning curve and had to adapt their styles and 
approaches as the course progressed. The established teaching teams tended to have clearly 
defined roles and strong working relationships. 


Administrative data show that more than 70 percent of treatment group members participated 
in at least one I-BEST course. Among those who participated, 58 percent of treatment group 
members received some type of credential, mostly commonly a workforce credential requiring 
20 more credits; 63 percent of the treatment group participated in additional college classes 
beyond I-BEST (with substantial differences across colleges and occupational programs). The 
high rate of progression to additional education and training (even in the absence of credential 
receipt) indicated some I-BEST students did not complete all of the course requirements for a 
given I-BEST program, but still continued on to more advanced or perhaps different courses of 
study. Interviews conducted for the study indicated some students may not have completed all 
the requirements to obtain the college-issued credential, but may have met the course 
prerequisites for proceeding to the next level of training without the short-term credential. 
Staff also reported that some students changed their focus or interests while attending I-BEST 
and thus continued in a different course of study without completing I-BEST. In addition, 
because of college staff’s perceptions of the local labor market, some programs de-emphasized 
obtaining the short-term, college-issued certificates after |-BEST. In such cases, the focus was 
instead on progression to higher levels of training or degree programs, even if it meant not 
obtaining the shorter term credential. 
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I-BEST in the three colleges had significant effects on participants’ receipt of education and 
training, as provided by the 18-month follow-up survey. The I-BEST program produced a 23 
percentage point difference in the proportion of treatment group members who received 
training in any subject compared with the control group (76 percent versus 53 percent). There 
was a similarly large effect on enrollment in two-year colleges overall. Treatment group 
members also received increased advising and supports at their colleges, including career 
counseling; help arranging supports for school, work, or family; and receipt of job search or 
placement services. Though these impacts were 13 or 14 percentage points, the overall 


proportion of treatment group members who reported receiving these services was still 
relatively low. 
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5. Early Impacts of I-BEST 


This chapter reports estimates of I-BEST’s early impacts on educational attainment, career 
progress, and a set of non-economic outcomes across the three study community colleges. The 
main estimates are based on administrative data and cover impacts over a 24-month period 
after random assignment ended for the full sample of 632 study participants. Additional 
estimates are based on data from the 18-month follow-up survey, which had a response rate of 
73 percent. 


The chapter begins by describing hypothesized impacts and the various outcomes that the team 
analyzed. Subsequent sections in this chapter present findings on education, career progress, 
and non-economic outcomes, respectively. Each of these sections distinguishes between 
confirmatory, secondary, and exploratory analyses. 


5.1. Key Hypotheses and Outcomes 


Through the combination of team teaching strategies that paired basic skills instructors and 
occupational training instructors for at least 50 percent of occupational training class time, 
support classes, advising, and financial assistance, I-BEST aimed to increase enrollment, credit 
attainment, credential completion, and career knowledge. In the longer term, I-BEST positioned 
students for more advanced credentials and career-track employment. Each I-BEST program 
was part of a career ladder, and students could exit and return or continue their education. 
Subsequent reports will focus on additional education and employment outcomes. 


The research team delineated outcomes as confirmatory, secondary, and exploratory. Exhibit 
5-1 lists and describes each outcome. 


The confirmatory outcome in the I-BEST analysis is the tota/ number of academic and workforce 
credits earned. Academic credits are attached to courses that are suitable for academic transfer 
to another college, such as English Composition | and Introduction to Statistics. Workforce 
credits are attached to training courses that are categorized as occupational in nature and 
include I-BEST courses such as Spreadsheets | and Applied Mechanics.** The research team 
determined that these two types of earned credits were the best educational indicator as to 
whether I-BEST was effective. The receipt of these credits reflects the ability of the treatment 


44 Approximately 93 percent of all I-BEST courses are categorized as occupational training courses according to 
SBCTC course records. The remaining I-BEST courses are categorized as basic skills, developmental, or 
academic courses. Among all courses taken by students in the sample, 87 percent of all academic courses 
were worth five credits, while the vast majority of workforce courses ranged from one to five credits, with five 
credits being the most frequent occurrence (31 percent). 
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group to engage in the array of academic and occupational activities that can support 
longer-term career success. (See Definitions of Types of Courses and Credentials box below.) 


Secondary analyses included tests of hypotheses for additional education outcomes, including 
certificate and degree completion, and enrollment in various categories of courses at SBCTC 
colleges, as well as separately delineated academic and workforce credits earned. Secondary 
analyses also included measures of career progress. 


Finally, exploratory analyses provided additional evidence on the program’s impacts ona 
number of educational and social outcomes. Educational outcomes include measures at six- 
month intervals after random assignment of whether students continued to enroll in basic skills 
and developmental courses, whether students enrolled specifically in developmental math and 
reading courses, and whether they completed college-level coursework in algebra or English 
(“gatekeeper” courses).* As I-BEST is a diversion for many students who otherwise would have 
faced basic skills or development courses that do not result in college credit, these tests will 
examine whether I-BEST diminished flows into this coursework, and instead diverted students 
into college-level coursework. Additionally, the research team expected I-BEST to have some 
impact on insights, habits, and functioning in a number of domains, including perceived 
measures of psycho-social skills, life stressors, and family structure. 


Exhibit 5-1: Outcomes in the Impact Analysis 


Sample Size“ 
Outcome Operationalization Source Treatment Control 
Total number of academic and ~— Within 24 months of random SBCTC records 315 316 
workforce credits earned assignment 
Received a certificate or Within 24 months of random SBCTC records 315 316 
degree from a college assignment 
Enrolled in college academic Earned within: Months 1-6; SBCTC records 315 316 
courses, occupational training Months 7-12; Months 13-18; 
courses, and any college Months 19-24; Any month 
courses 
Cumulative number of Earned by end of: Month 6; SBCTC records 315 316 
academic credits, workforce Month 12; Month 18; Month 24 
credits, total academic and 
workforce credits 
Received an occupational Within 18 months of SBCTC records, SBCTC: 315 SBCTC: 316 
credential randomization PACE short-term Survey: 238 Survey: 225 
follow-up survey 


45 Developmental math and reading courses, as well as college-level algebra and English courses, were flagged 
using specific Classification of Instructional Programs (CIP) codes in this analysis to distinguish the courses that 
are traditionally categorized as being in the remedial course sequence and “gatekeeper” course framework. 

46 One student who was randomized to the control group left the sample at the time that outcomes were 
measured. 
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Perceived career progress 


Confidence in career 
knowledge 


Access to career supports 


Working in a job paying at or 
above specified wage 
Working in a job requiring at 
least mid-level skills 


Enrolled in basic skills or 
developmental education 
courses 

Enrolled in developmental any 
math course 

Enrolled in any developmental 
reading/writing course 
Enrolled in and completed 
college-level algebra course 
Enrolled in and completed 
college-level English course 


Grit 


Academic self-confidence 


Core self-evaluation 


Abt Associates 


3-item scale of self-assessed 
Career progress; response 
categories range from 
1=strongly disagree to 
4=strongly agree 

7-item scale of self-assessed 
career knowledge; response 
categories range from 
1=strongly disagree to 
4=strongly agree 

7-item scale counting number 
of types of career-supportive 
relationships in workforce and 
education settings; ranging 
from 0 to 7 possible types; 
response categories are 0=no 
or 1=yes 


Earning $12/hour or more@ 


Employed in a job requiring 
calibrated set of skills based 
on federal standards 


Enrolled in: Months 1-6; 
Months 7-12; Months 13-18; 
Months 19-24; Any month 


8-item scale capturing 
persistence and determination; 
response categories range 
from 1=strongly disagree to 
4=strongly agree 

12-item scale; response 
categories range from 
1=strongly disagree to 
6=strongly agree 

12-item scale; response 
categories range from 
1=strongly disagree to 
4=strongly agree 


PACE short-term 
follow-up survey 


PACE short-term 
follow-up survey 


PACE short-term 
follow-up survey 


PACE short-term 
follow-up survey 
PACE short-term 
follow-up survey 


SBCTC records 


SBCTC records 
SBCTC records 
SBCTC records 


SBCTC records 


Exploratory Outcomes: Psycho-Social Skills, Life Stressors, and Family Structure 


PACE short-term 
follow-up survey 


PACE short-term 
follow-up survey 


PACE short-term 
follow-up survey 


238 


238 


238 


238 


238 


315 


315 


315 


315 


315 


238 


238 


238 


Sample Size“ 


225 


225 


225 


Secondary Outcomes: Employment Success 


225 


225 


Exploratory Outcomes: Education Success 


316 


316 


316 


316 


316 


225 


225 


225 


Outcome Operationalization Source _ Treatment Control 
Secondary Outcomes: Self-Assessed Career Development 
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Sample Size“ 
Outcome Operationalization Source Treatment Control 
Social belonging in school 5-item scale capturing sense of PACE short-term 238 225 
belonging; response follow-up survey 
categories range from 
1=strongly disagree to 
4=strongly agree 
Financial hardship 2-item scale reported as either | PACE short-term 238 225 
an inability to pay follow-up survey 
rent/mortgage or not enough 
money to make ends meet; 
response categories are O=no 
and 1=yes 
Life challenges 7-item scale capturing life PACE short-term 238 225 
challenges that interfere with follow-up survey 
school, work, or family 
responsibilities; response 
categories range from 1=never 
to 5=very often 
Perceived stress 4-item scale capturing PACE short-term 238 225 
perceived stress; response follow-up survey 
categories range from 1=never 
to 4=very often 
Living with spouse Response categories are 0=no ~~ PACE short-term 238 225 
and 1=yes¢ follow-up survey 
Had child since random Response categories are 0=no ~~ PACE short-term 128 144 


assignment / currently pregnant 


and 1=yes¢ 


follow-up survey 


4 Threshold selected because it was close to the 60th percentile of hourly wages among employed control group members. 

> Skill levels based on the federal O*NET system, with thresholds targeted to PACE program target occupations. Occupational categories 
were coded for PACE by Census Bureau staff from standard open-ended survey items. 
© Living with an unmarried partner is not counted as living with a spouse. 


4 Asked of women only. 
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Definitions of Types of Courses and Credentials 
Courses: 
e Academic: Credit-bearing courses that are transferrable to four-year colleges, and are not considered 
basic skills or developmental. Most general education requirements fit this category. 
e Workforce: Credit-bearing occupational training courses that focus on specific technical skills. 


e Basic skills: Non-credit Adult Basic Education courses, GED coursework, and English as a Second 
Language courses. 


e Developmental education: Also known as pre-college or remedial education; non-credit courses 
outside of the basic skills courses listed above, that are specifically designed to prepare students for 
college-level algebra and English courses. 


e College-level algebra and English: Academic courses that are designated as having a course number of 
100 or above with specific Classification of Instructional Programs (CIP) codes of 23 (algebra) and 27 
(English). 


Credentials: 


e Workforce award: A certificate that requires 20 or more credits to complete (e.g., Vehicle Service 
Technician Certificate) or an Associate’s in Applied Science (AAS) degree (e.g., AAS in Welding 
Technology). The AAS is not fully transferrable to a four-year bachelor’s program. 


e Workforce completion: A certificate that requires less than 20 credits to complete. 


e = Associate’s of Arts (AA) or Associate’s of Science (AS): Degrees that usually require two years of 
coursework, and are usually transferrable to any public university, as well as to many private 
universities. 


5.2. Impacts on Educational Attainment 


This section presents impact estimates for key measures of educational progress for the I-BEST 
sample, aggregated across the three colleges. It first highlights the confirmatory test to 
determine whether early impacts are on the expected path. Next, it examines findings for the 
secondary and exploratory education outcomes. 


e |-BEST increased the total number of academic and workforce credits earned at colleges 
(confirmatory hypothesis). 


Exhibit 5-2 shows that I-BEST had a 13-credit impact on academic and workforce credits earned. 
Over the 24-month period, treatment group members completed 24 credits compared with 11 
credits for the control group. Both types of credits are college-level credits (i.e., non-remedial 
and applicable towards credentials), and the accumulation of these credits is a positive 
indicator of academic progress, either towards a credential or, for academic credits, towards 
transferring to four-year programs in the future. The majority of the credits (84 percent) earned 
were workforce credits.*” 


47 The average academic and workforce credits received by treatment students is 24.3 (see Exhibit 5-2) and 20.5 


of those credits are workforce credits. The same figures for control group students are 11.2 and 8.2, 
respectively (73 percent). 


‘Abt Associates 5. Early Impacts of I-BEST I pg. 62 


Washington State’s Integrated Basic Education and Skills Training (I-BEST) Program in Three Colleges 
Implementation and Early Impact Report 
PACE 


This finding demonstrates that students with measurably low skills can earn college-level 
credits through I-BEST. A closer look at enrollment (the top panel of Exhibit 4-4) shows that the 
treatment/control difference in credits earned is driven primarily by more treatment group 
than control group members enrolling in college. 


Exhibit 5-2: Early Impacts on Education Outcomes (Confirmatory Hypothesis) 


: Treatment | Control Impact Standard Effect 
Cornmmate ry cutcome Group Group | (Difference) Error Size Bvalue 
Total number of academic and 24.3 11.2 +13. 15% 2.2 +0.47 <.001 
workforce credits earned at 
colleges within 24 months 
Sample size 315 316 


SOURCE: Abt Associates calculations based on data from SBCTC records. 
NOTES: Statistical significance levels, based on one-tailed t-tests tests of differences between research groups, are summarized as 
follows: *** statistically significant at the one percent level; ** at the five percent level; * at the 10 percent level. 


e The I-BEST program increased credential completion, particularly completion of workforce awards. 


Beyond earning credits, a key I-BEST objective is for students to obtain a credential. Community 
college completion rates, even for programs that result in a short-term certificate, ** are 
typically low (Shapiro et al. 2014).*° Many students have competing demands in their lives, and 
finishing a program with a credential is a goal that can be derailed by various circumstances. 


With this in mind, it is particularly encouraging to see I-BEST increase the completion of any 
credential within the 24-month follow-up period by 32 percentage points, from 12 percent to 
44 percent (Exhibit 5-3). The vast majority of the students completed a workforce award, which 
could include an AAS degree. I-BEST increased the share of study participants who received 
workforce awards from eight percent to 41 percent. I-BEST also increased the attainment of the 
workforce completion credential by four percentage points, from five percent to nine percent. 
The small share of study participants who completed this last certificate reflects the fact that 
I-BEST primarily provided programs requiring more than 20 credits to obtain a credential. 


I-BEST did not have an impact on completion of associate’s degrees transferrable to a four-year 
college. This is not surprising, as such degrees usually take more than two years to complete— 
longer than the follow-up period for this report. In addition, completion of a transferrable AA or 
AS degree was not an intended short-term outcome of I-BEST.”° 


48 Defined as any non-degree credential that takes less than one year of full-time study. 


49 A 2014 study from the National Student Clearinghouse finds that students who started at a two-year public 
institution had two-year award completion rates of about 30 percent within eight years. Approximately 56 
percent of students who started at a four-year public institution received an award within eight years. 

50 This is in contrast to the AAS degrees, which are non-transferrable to four-year bachelor’s programs and are 
often the next step in the pathway of several of these programs. 
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The impact on credentials completed was driven primarily by a higher completion rate among 
treatment group members enrolling in college than among enrolling control group members. 
That is, among students who enrolled in any course, 49 percent of treatment group members 
received a credential from the college (44 percent divided by 90 percent) versus 18 percent of 
control group members (12 percent divided by 68 percent). This difference of 31 percentage 
points for enrolled students is comparable to the impact on credential receipt reported above 
for the full sample. Thus, it appears that the I-BEST program was more effective in engaging 
students in programming that resulted in a credential than were other education and training 
opportunities in the community. 


The 18-month follow-up survey results on occupational credential receipt (the bottom panel of 
Exhibit 5-3) show that in addition to its impact on receiving a certificate from a SBCTC college 
(31 percentage points), |-BEST had a 15 percentage point impact on receiving an occupational 
credential from a licensing or certification agency. Almost a third of the treatment group (32 
percent) received such a credential, compared with 17 percent of the control group. Combining 
credential receipt from a college and a licensing or certification agency, almost three-quarters 
of the treatment group received a credential within 18 months of random assignment, 
compared with less than a quarter of the control group. 


Exhibit 5-3: Early Impacts on Education Outcomes (Secondary Hypotheses) 


Treatment Control Impact Standard _ Effect 
cutcome Group Group (Difference) _—Error Size PAS 
Received a certificate or degree from a 
college within 24 months (N=631) 
Workforce completion? 8.5 47 +3.0°* 1.9 +0.16 026 
Workforce award? 41.3 7.9 +33.3*** 3.1 +0.86 <.001 
Associate’s degree or higher 0.3 0.3 0.0 0.4 -0.01 535 
Any credential 44.2 12.0 +32:1* 3.3 +0.78 <.001 
Enrolled in college (%) (N=631) 
Academic courses 
Months 1-6 14.7 15.2 -0.5 2.8 -0.01 570 
Months 7-12 16.0 14.9 +11 2.8 +0.03 347 
Months 13-18 14.2 11.4 +2.8 2.6 +0.09 142 
Months 19-24 10.0 7.0 +3.0* 2.2 +0.11 .090 
Any month 31.8 25.6 +6.1* 3.5 +0.14 041 
Occupational training courses 
Months 1-6 73.3 21.5 +51.7*" 3.4 +1.20 <.001 
Months 7-12 40.0 21.2 +18.8** 3.6 +0.42 <.001 
Months 13-18 22.5 17.1 +5,4** 3.1 +0.14 041 
Months 19-24 18.6 16.5 +2.2 3.0 +0.06 237 
Any month 80.5 39.6 +40.9** 3.6 +0.91 <.001 
Any courses¢ 
Months 1-6 85.3 58.2 42/1" 3.4 +0.63 <.001 
Months 7-12 52.5 40.2 412.3" 3.9 +0.25 <.001 
Months 13-18 36.6 29.1 +7.5"* 3.7 +0.16 .022 
Months 19-24 27.7 22.8 +5.0* 3.4 +0.11 075 
Any month 89.6 67.7 +21.9*" 3.2 +0.55 <.001 
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Treatment Control Impact Standard _ Effect 
outcome Group Group (Difference) — Error Size PNG 
Cumulative credits (N=631) 
Academic credits through 
Month 6 0.9 0.8 +0.1 0.2 +0.05 .263 
Month 12 18 1.7 +0.1 0.4 +0.02 389 
Month 18 2.9 2.5 +0.4 0.5 +0.06 239 
Month 24 3.8 3.0 +0.7 0.7 +0.09 129 
Workforce credits through 
Month 6 6.9 1.4 +5,5%* 0.5 +0.92 <.001 
Month 12 13.3 3.7 +9.6"** 1.0 +0.73 <.001 
Month 18 16.8 6.0 +10.8*** 15 +0.57 <.001 
Month 24 20.5 8.2 +12.4** 2.0 +0.49 <.001 
Academic and workforce credits 
through 
Month 6 78 2.1 +577" 0.5 +0.87 <,001 
Month 12 15.1 5.4 +9,7*** 11 +0.68 <.001 
Month 18 19.7 8.5 +11,.2"* 17 +0.53 <.001 
Month 24 24.3 11.2 +13.1* 2.2 +0.47 <.001 
Received an occupational credential 
through 18 months from 
A college (%) (N=631) 40.6 9.8 +30.8*** 3.1 +0.78 <.001 
Other place (%) (N=463) 1.0 43 -3.3 1.4 -0.22 .989 
Licensing or certification agency 32.0 16.9 15.144 3.8 +0.37 <.001 
(%) (N=463) 
Any place/authority (%) (N=463) 51.5 23.9 +27.6*** 4.2 +0.60 <.001 
Sample size 315 316 


SOURCE: Abt Associates calculations based on data from SBCTC records and from the PACE short-term follow-up survey. 
NOTES: Statistical significance levels, based on one-tailed t-tests tests of differences between research groups, are summarized as 
follows: *** statistically significant at the one percent level; ** at the five percent level; * at the 10 percent level. 

4 Certificates requiring 1-19 credits. 

> Certificates requiring 20 or more credits, including Associate's in Applied Sciences (AAS) degrees. 

¢ Any academic, workforce, basic skills, or developmental courses. 


The size of the impact on credential receipt is similar or larger than the impact estimates from 
previous studies based on quasi-experimental designs (Jenkins, Zeidenberg, and Kienzl 2009; 
Zeidenberg, Cho, and Jenkins 2010). That the large impact sizes hold after using a rigorous 
research design provides strong evidence of a causal link between access to I-BEST and 
credential receipt. 


e Includes /-BEST substantially increased enrollment in college courses, driven primarily by 
enrollment in occupational training. 


The research team examined I-BEST’s impact on enrollment in academic and occupational 
training courses within 24 months and by six-month intervals. The latter was intended to 
determine how the program’s impact changes over time. The results indicate that for academic 
course enrollment, there was a cumulative six percentage point increase in enrollment over the 
24 months (see Exhibit 5-3). This suggests that while I-BEST offered occupational training, it still 
spurred academic course enrollment among this population of disadvantaged students. Exhibit 
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5-4 shows academic course enrollment for each six-month period, in which the impact 


increases over time. 


Exhibit 5-4: Academic Course Enrollment over Time 
30% 
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However, the majority of students pursued occupational training. I-BEST increased enrollment 
in credit-bearing occupational courses by 41 percentage points (Exhibit 5-5). This makes sense, 
given that all I-BEST courses fall into this workforce category, and most treatment group 
students enrolled in I-BEST courses. About four in five treatment group members (80 percent) 
enrolled in occupational training courses over the 24-month period (Exhibit 5-3), compared 
with 40 percent in the control group. Exhibit 5-5 shows enrollment in six-month increments. 
The impacts are particularly strong in months one through six (52 percentage points), indicating 
that more students in the treatment group enrolled in occupational courses quickly, compared 
with their control group counterparts. 
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Exhibit 5-5: Occupational Training Course Enrollment over Time 
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Overall, when accounting for all courses in college, including basic skills and developmental 
courses, I-BEST increased enrollment by 22 percentage points within the 24 months following 
random assignment (Exhibit 5-3). Almost 90 percent of the treatment group enrolled in some 
coursework within this timeframe, compared with 68 percent of the control group. 


e  |-BEST’s impact on cumulative credits increased over time, though the difference in enrollment 
between treatment and control declined. 


As noted earlier, I-BEST increased completion of academic and workforce credits over the 24 
month follow-up. This impact was driven by enrollment in and completion of occupational 
courses, as there is no overall impact on credits earned in academic courses. Over time, the gap 
in credit completion grew between the treatment and control groups. As the third panel in 
Exhibit 5-3 shows, through month six, the impact on cumulative workforce credits was about six 
credits; the impact increased to about 10 credits by month 12, 11 credits by month 18, and 12 
credits by month 24—all results are statistically significant at the one percent level. 


Conversely, I-BEST’s impact on course enrollment faded over time, as seen in the second panel 
of Exhibit 5-3. The impact on enrollment in any course was at its highest in the first six months 
at 27 percentage points, and dropped to five percentage points by months 19 through 24 
(statistically significant at only the 10 percent level). A possible explanation for the decline in 
impact size over time is that I-BEST has a strong focus on course enrollment at the front end. 
Another explanation is that |-BEST students may have completed their courses faster than their 
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control group counterparts. The control group may have taken longer to complete the 
necessary developmental or ABE coursework to get into college-level coursework, thereby 
depressing the enrollment impact over time. Declining impacts on course enrollment and 
increasing impacts on course completion over time, with positive impacts on credentials 
received, is what we might expect from a career pathways program designed to provide initial 
training that leads to employment. 


e The I-BEST program’s impact on enrollment in basic skills and developmental courses faded over 
time. Though there was no impact on developmental course enrollment by subject, there were 
positive impacts on both enrollment in and completion of college-level algebra and English 
courses. 


The results in the first panel of Exhibit 5-6 indicate that there was a 19 percentage point 
increase in the enrollment of basic skills and developmental education courses in the first six 
months after randomization.*! This impact, however, faded over the next 18 months. There 
was a sizeable drop-off in impact in months seven through 12 (six percentage points, 
statistically significant at only the 10 percent level), and no statistically significant impacts over 
the next 12 months. Over the course of the entire 24 months, there was a 21 percentage point 
impact on enrollment in these courses. Much of the impact in the enrollment of these types of 
courses, particularly early on, is attributable to basic skills support courses required for I-BEST 
certificates. °? 


To better discern any possible impacts of I-BEST specifically on developmental coursework, the 
research team also examined separately enrollment in developmental math courses and in 
developmental reading courses. The team separated out these two courses because the 
enrollment and completion of these two types of courses can often differ from each other. As 
shown in panel two, the findings suggest that most of the positive impacts in the first six 
months are likely due to basic skills course enrollment, as there is a lack of impact on 
developmental math and reading course-taking across all time periods. 


However, within the 24-month follow-up period, I-BEST increased enrollment in and 
completion of college-level math (algebra) and English—credit-bearing academic courses with 
course numbers of 100 or above. For algebra, I-BEST increased enrollment by 10 percentage 
points and completion by nine percentage points; for English, I-BEST increased enrollment by 


51 In considering basic skills courses, the research team chose to analyze course enrollment and did not measure 


credit attainment. Basic skills courses do not count towards the course requirements for credential 
attainment. Though the colleges attached credits to basic skills courses for their own institutional purposes, 
we do not analyze basic skills credit attainment, but instead focus on enrollment in basic skills courses. 


52 See Chapter 3 for the courses required for specific |-BEST programs across the three colleges. Almost all 


programs have a basic skills course requirement, which is why there is a high percentage of students enrolling 
in basic skills or developmental courses using course transcript information. 


Abt Associates 5. Early Impacts of I-BEST | pg. 68 


Washington State’s Integrated Basic Education and Skills Training (I-BEST) Program in Three Colleges 
Implementation and Early Impact Report 
PACE 


nine percentage points and completion by six percentage points. These findings (shown in the 
third panel of Exhibit 5-6) are encouraging in terms of advancement in college coursework, 
because these algebra and English courses serve as prerequisites for many other courses and 
are often easily transferrable to four-year colleges. 


Exhibit 5-6: Early Impacts on Other Education Outcomes (Exploratory Hypotheses) 


Treatment Control Impact Standard _—_ Effect 
asl Group Group _ (Difference) _Error Size aba aes 
Enrollment in basic skills or developmental 
courses in (%) 
Months 1-6 64.7 45.9 +18.8** 3.9 +0.38 <,001 
Months 7-12 29.2 22.8 +6.4* 3.4 +0,15 .061 
Months 13-18 14.6 12.7 +2.0 2.7 +0.06 71 
Months 19-24 11.6 8.2 +3.4 2.4 +0,11 151 
Any month 73.2 51.9 +21.3*" 3.8 +0.45 <.001 
Enrollment in developmental courses over 
time (%)@ 
Enrolled in any developmental math 
course in: 
Months 1-6 5.1 6.6 -15 1.9 -0.07 414 
Months 7-12 5.5 6.6 -11 1.8 -0.05 548 
Months 13-18 6.0 3.8 +2.2 1.7 +0.10 .200 
Months 19-24 5.6 3.2 +2.4 1.6 +0,12 145 
Any month 145 13.0 +15 2.7 +0.05 565 
Enrolled in any developmental 
reading/writing course in 
Months 1-6 46 6.3 -17 1.8 -0.07 348 
Months 7-12 3.7 3.8 -0.1 15 -0.01 948 
Months 13-18 41 2.8 +13 15 +0.07 381 
months 19-24 2.4 1.9 +0.5 1.2 +0.03 675 
Any month 12.3 12.0 +0.3 2.6 +0.01 913 
Enrollment and completion of college-level 
academic courses (%) 
Enrolled in 
Introductory college-level math 22.0 11.7 +10.3*** 2.9 +0.28 <.001 
Introductory college-level English 25.5 16.8 +8,7*** 3.1 +0.22 .006 
Completed with a grade of C or better: 
Introductory college-level math 19.4 10.1 +9,.3*** 2.7 +0.27 <.001 
Introductory college-level English 20.7 14.2 +6,.4** 2.9 +0.17 .029 
Sample size¢ 315 316 


SOURCE: Abt Associates calculations based on data from SBCTC records. 

NOTES: Statistical significance levels, based on two-tailed t-tests tests of differences between research groups, are summarized as follows: 
*** statistically significant at the one percent level; ** at the five percent level; * at the 10 percent level. 

4 Developmental coursework enrollment was derived using administrative data. Courses were flagged developmental if they had a course 
number below 100. 

> Also referred to as “gatekeeper” courses, these are the first college-level courses that many community college students take as 
prerequisites for higher-level courses. 

¢ Sample size is 315 treatment cases and 316 control cases for all analyses shown in this table. Completion of introductory college-level 
math or English course is not conditional on enrollment. 
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5.3. Impacts on Early Career Progress (Secondary Hypotheses) 


This section presents impact estimates for employment outcomes and self-assessments of career 
progress. 


e |-BEST improved perceived career progress, but there was no evidence of impact on skilled 
employment. 


The estimates in Exhibit 5-7 show a positive impact on perceived career progress, but no impacts on 
the other assessments of career progress in the first 18 months. The results also show no evidence 
of impacts for the other two indicators of career pathways employment: working in a job (1) 
earning more than $12 per hour and (2) that requires at least mid-level skills. 


Exhibit 5-7: Early Impacts on Selected Career Outcomes (Secondary Hypotheses) 


Treatment Control Impact Standard Effect 
Catcome Group Group (Difference) Error Size Ee 
Indices of Self-Assessed Career 
Development (average): 
Perceived career progress? 3.32 3.19 +0.13** 0.08 +0.17 043 
Confidence in career knowledge? 3.14 3.08 +0.06 0.05 +0.10 .136 
Access to career supports* 1.72 1.77 —0.04 0.03 -0.14 .936 
Indicators of Career Pathways 
Employment (%): 
Working in a job paying $12/hour 23.0 23.8 -0.8 4.1 -0.02 516 
or more 
Working in a job requiring at least 5.1 9.5 -4.4 2.6 -0.16 .953 
mid-level skills? 
Sample sizee 238 225 


SOURCE: Abt Associates calculations based on data from the PACE short-term follow-up survey. 

NOTES: Statistical significance levels, based on one-tailed t-tests tests of differences between research groups, are summarized as follows: *** 

Statistically significant at the one percent level: ** at the five percent level; * at the 10 percent level. 

4 Three-item scale tapping self-assessed career progress; response categories range from 1=strongly disagree to 4=strongly agree. 

> Seven-item scale of self-assessed career knowledge; response categories range from 1=strongly disagree to 4=strongly agree. 

© Seven-item scale of types of career-supportive relationships in workforce and education settings, ranging from zero to seven possible types; 

response categories are either 1=no or 2=yes. 

4 Whether employed in a job requiring calibrated set of skills based on federal standards. 

€ Sample sizes in this row apply to sample members responding to the PACE follow-up survey. 
Given I-BEST’s short-term focus on building basic and occupational skills, it is not surprising that the 
effects for perceived career progress are more positive than actual employment outcomes at 24 
months. Though the observed progress in training might foster participants’ sense of career 
progress, the fact that more treatment than control group members continued to be enrolled in 
training programs at the end of the 18-month survey suggests that it may be too early to expect 
unambiguously positive impacts on actual attainment of career-path jobs. Whether I-BEST helps 


participants obtain career path jobs in the longer term will be addressed in a later report. 


5.4. Impacts on Psycho-Social Skills, Life Stressors, and Family Structure 


The large positive impacts on course enrollment and credential completion, as well as the positive 
impact on self-assessed career progress, suggest possible positive effects on other self-assessed 
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outcomes associated with college success. However, individuals in a program that stresses psycho- 
social of skills may come to have higher expectations of their own performance than control group 
members, rating the same objective level of performance more negatively (Duckworth and Yeager 
2015). This potential for measurement bias means the predicted direction of effect is unclear. As 
such, the study treats these analyses as exploratory (i.e., subject to two-sided tests). 


e __I-BEST slightly increased social belonging in school and decreased perceived stress. 


Results in Exhibit 5-8 show a positive effect of I-BEST on social belonging in school, with treatment 
group students reporting a slightly higher level of belonging on a four-point scale (statistically 
significant at the 10 percent level). 


Exhibit 5-8: Early Impacts on Other Outcomes (Exploratory Hypotheses) 


Treatment Control Impact Standard Effect 
Ourcaine Group Group (Difference) Error Sie Peer 
Indices of Psycho-Social Skills 
(average): 
Grit@ 3.03 2.99 +0.03 0.04 +0.07 434 
Academic self-confidence? 4.49 4.48 +0.01 0.07 +0.02 842 
Core self-evaluation¢ 3.20 3.14 +0.06 0.04 +0.14 150 
Social belonging in school¢ 3.25 3.15 +0.09* 0.05 +0.18 054 
Indices of Life Stressors (average): 
Financial hardship® 0.63 0.64 -0.01 0.04 -0.03 .783 
Life challenges‘ 1.52 1.55 -0.04 0.05 -0.07 423 
Perceived stress9 2.15 2.29 -0.14** 0.07 -0.19 043 
Family Structure (%): 
Living with spouse 35.9 33.2 +2.7 3.5 +0.07 A444 
Had child since random 9.7 13.7 -4.0 3.6 -0.14 .266 
assignment/currently pregnant 
Sample size 238 225 


SOURCE: Abt Associates calculations based on data from the PACE short-term follow-up survey. 

NOTES: Statistical significance levels, based on two-tailed t-tests tests of differences between research groups, are summarized as follows: 
*** statistically significant at the one percent level; ** at the five percent level; * at the 10 percent level. 

4 Eight-item scale capturing persistence and determination, response categories range from 1=strongly disagree to 4=strongly agree. 

> Twelve-item scale; response categories range from 1=strongly disagree to 6=strongly agree. 

° Twelve-item scale; response categories range from 1=strongly disagree to 6=strongly agree. 

4 Five-item scale capturing sense of belonging; response categories range from 1=strongly disagree to 4=strongly agree. 

®Two-item scale capturing financial hardship, reported as either an inability to pay rent/mortgage or not enough money to make ends meet; 
response categories are 0=no and 1=yes. 

Seven-item scale capturing life challenges that interfere with school, work, or family responsibilities; response categories range from 
1=never to 5=very often. 

9Four-item scale capturing perceived stress; response categories range from 1=never to 4=very often. 

h Sample sizes in this row apply to sample members responding to the PACE follow-up survey. 


I-BEST decreased perceived stress, with I-BEST students reporting a lower level of stress. There 
were no significant impacts on the remaining indices of psycho-social skills or life stressors. There 
were also no significant impacts on family structure. 


These early findings suggest that in light of the financial and academic supports provided 
through I-BEST, the program may reduce stress among program participants and give them a 
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greater sense of social belonging. Whether these impacts persist, or even increase, as 
participants transition to employment is an open question to be addressed in later reports. 
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6. Conclusions 


Since its inception, I-BEST has garnered the attention of educators and researchers as a 
promising model to support educational and occupational advancement for students with low 
basic skills. The I-BEST program grew out of a concern that adult basic skills students were often 
not advancing beyond basic skills classes to college-level occupational programs, and therefore 
were not earning credentials. To better serve this low-skilled population, the I-BEST program 
modified instructional staffing, pairing basic skills instructors and occupational instructors for 
team teaching in occupational training courses. The I-BEST programs evaluated here also 
assisted students with school-related expenses and offered advising support. Without I-BEST, 
students whose college entrance test scores were too low for them to enroll directly in their 
desired occupational training program would have had to successfully complete remedial 
classes to increase their basic skills to the required levels. 


This study found that the evaluated I-BEST programs were implemented largely as designed, 
with various forms of team teaching, as well as financial and advising supports delivered to a 
population with low education levels (about 30 percent did not have a high school diploma or 
GED). The program substantially increased enrollment in credit-bearing courses, driven 
primarily by students’ enrollment in occupational training classes. It also increased treatment 
group members’ receipt of career counseling and other supports. Importantly, I-BEST programs 
also produced increases in the number of academic and workforce credits earned and the 
percentage of students who had earned a workforce credentials within the 24-month follow-up 
period. This concluding chapter examines the implications of these early results for policy and 
practice and looks ahead to questions the evaluation will address in the longer term. 


6.1. Implications of |-BEST Programs 


Because of the interest at the federal, state, and local levels in I-BEST as a strategy to improve 
education and employment outcomes for low-skilled adults, replications of the model are 
already underway in several states and localities. The PACE results, which are the first from an 
experimental evaluation of the I-BEST model, have a number of implications for further 
development of related initiatives. 


e Earlier non-experimental research showed the potential of I-BEST to increase academic success for 
low-skilled adults. Initial findings from the PACE evaluation provide strong causal evidence of the 
effect the I-BEST program can have on education outcomes. 


The PACE evaluation provides the first rigorous evidence on I-BEST from a range of occupational 
programs at three of Washington State’s 34 community and technical colleges. Though not 
representative of all |-BEST programs statewide, these results greatly strengthen prior evidence 
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(e.g., Zeidenberg, Cho, and Jenkins 2010) that I-BEST can produce increases in receipt of credits 
and credentials. In particular, the results demonstrate that some students who may not initially 
qualify for college-level programming because of low basic skills levels can complete college- 
level work when additional basic skills instruction, financial assistance, and advising supports 
are built into the occupational training. Future reports will provide information on I-BEST’s 
employment effects. Longer follow-up, planned as part of the PACE study, is needed to 
determine I-BEST’s effects on participants’ economic outcomes. 


e Though the I-BEST programs themselves last one to two quarters, many students used them as a 
launching pad for additional college-level education and training. 


A high proportion of treatment group members who attended I-BEST programs (63 percent) 
also enrolled in additional college classes. On average, those who attended an I-BEST program 
were enrolled in college courses for four quarters—longer than the I-BEST portion of the 
occupational training lasted. Approximately one-quarter were still enrolled in college at the end 
of the 24-month follow-up period. In addition, the impact on earning credentials was driven 
primarily by the higher rate of completion among treatment group members who enrolled in 
college than among the control group. Thus, it appears that the I-BEST program was more 
effective in engaging students in programming that resulted in a credential than other 
education and training programs were. 


e For students interested in pursuing higher levels of education, it may be important to maintain 
connections between occupational and academic programs. 


Most of the credits and credentials completed by I-BEST participants were from workforce 
courses, meaning they are not transferable to four-year colleges. But there is evidence that 
some students also incorporate more academic coursework in their education. By design, 
workforce credits are earned in occupational courses that focus on specific technical skills and 
do not include academic credits that are generally transferable to four-year colleges (and that 
can be used to meet general education requirements for higher-level degrees). However, 
primarily through enrollment in courses after I-BEST, the program also increased enrollment in 
and completion of college-level algebra and English, both of which are credit-bearing academic 
courses. These findings are encouraging for potential advancement in college coursework, 
because these introductory math and English courses serve as prerequisites for many other 
college courses and are often transferrable to four-year colleges. Some I-BEST instructors were 
proactive in encouraging this connection to higher-level education courses, particularly if there 
were limited job advancement opportunities for program completers with short-term 
workforce credentials. However, whether and when this transition from workforce to academic 
credits is appropriate is an important consideration when designing career pathways beyond 
the initial |-BEST programming. 
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e Based on discussions with I-BEST staff, team teaching was most successful when co-instructors had 
planning time and worked together over multiple quarters. 


Although team teaching was used across all the programs in the study, I-BEST staff reported 
variations in how it was operationalized and challenges the teams faced. The I-BEST program 
represents a departure for many occupational instructors, who are used to being alone ina 
classroom and teaching students with higher academic skills. To effectively implement I-BEST, 
the instructors reported needing to adapt to the learning styles of students with lower basic 
skills. They also needed to develop instructional approaches that effectively integrated a 
second instructor. The implementation study underscores the value of professional 
development activities to help instructors define and integrate their instruction and the benefit 
of instructors in working together as team for more than one quarter. Several of the programs 
reported that part-time and adjunct staffing was a challenge to deeper integration. While 
common in community colleges, the use of part-time and adjunct staffing limited the planning 
time available to the teaching teams. 


e Although the I-BEST programs operated largely as designed, the implementation study identified 
areas for further attention and development. 


First, although the program sought to target low-skilled individuals, colleges had some difficulty 
recruiting students. In order to scale up, sites were required to modify their outreach 
strategies. Second, formalized connections between education and employment were not 
strong in most of the I-BEST programs. Though some students proceeded to additional 
coursework, strengthening this connection would likely benefit students who are focused on 
earning workforce credits and credentials. I-BEST occupational instructors, some of whom had 
industry experience, sometimes provided informal individualized job assistance, but this kind of 
connection to the labor market was not consistent across the programs. Finally, foundation 
funding during the PACE study enabled the colleges to implement additional resources for 
dedicated student advisors and “fill-the-gap” financial assistance; this support not typically 
available at in other I-BEST programs. Study results show increased receipt of these supports. 
This indicates supports may contribute to the observed program effects, and may be important 
for other program administrators to consider. 


6.2. What Lies ahead for the Evaluation of I-BEST 


This report finds that I|-BEST had a strong positive impact on college enrollment, credits earned, 
and credential receipt over a 24-month period. Though promising, longer-term follow-up on a 
broader range of outcomes, particularly employment and earnings, is needed to more 
definitively assess the I-BEST program: 


e Will I-BEST’s impact on educational attainment remain stable, increase, or decrease? 
In particular, it will be important to assess whether students receive additional 
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credentials, beyond those offered through the short-term I-BEST program. The positive 
early impacts might indicate permanent improvements of the treatment group relative 
to the control group. Conversely, control group members could catch up over time, so 
that their education outcomes are no longer significantly different from those of the 
treatment group. 


e Will I-BEST’s impact on educational attainment translate into impacts on employment 
and earnings? The 24-month analyses examined only a very few employment-related 
outcomes. Future analyses will assess the impact of I-BEST on a broader array of 
employment outcomes, including earnings, hourly wages, receipt of fringe benefits, and 
stability of employment. 


e Does I-BEST have other impacts on participants and their families? Key outcomes to 
address in the future include the effect of the program on individual and household 
income and material well-being. 


e Is I-BEST cost beneficial? Future analyses will explore the costs of the program relative 
to the benefits it produces for participants and society. 


The next PACE report on I-BEST will cover a 36-month follow-up period and provide a 
systematic look at impacts on employment for a period when any such impacts can be expected 
to emerge. That report will examine a broad variety of employment outcomes, including 
average employment and earnings over successive follow-up quarters, job characteristics (e.g., 
occupation, hourly wage rate, receipt of fringe benefits, career progress). Thus, the report will 
begin to answer whether the education and training experiences of I-BEST treatment group 
members will translate into economic gains in the workplace in the longer term. 


In addition, a later report will examine the long-term effects of I-BEST on earnings at 
approximately 72 months after random assignment. 
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